We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Lidar classification, accuracy and change detection using the Norwegian open lidar data archive.

00:00

Formal Metadata

Title
Lidar classification, accuracy and change detection using the Norwegian open lidar data archive.
Title of Series
Number of Parts
351
Author
Contributors
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language
Production Year2022

Content Metadata

Subject Area
Genre
Abstract
Three dimensional representations of surface terrain and structure is essential for a range of widespread applications and forms a base dataset that underlies many decision making processes. A few examples include land use planning, areal overview, operational analysis, emergency handling, route and transport planning, geographical and meteorological modelling etc. Recently, the Norwegian Government and the Norwegian Mapping Authority tasked the acquisition of high resolution Light Detection and Ranging (LIDAR) data covering the entire mainland with a minimum of 2 point measurements per meter. In addition, all aerial lidar acquisitions that were tasked by the government since the early 2000s are also publically available for download. In this work using FOSS, we discuss the height accuracy of ground classified datasets (i.e. Digital Terrain Models, Digital Surface Models) with varying original acquisition ground point densities. We create classification pipelines that allow us to calculate derivative products such as a “normalized” vegetation density and further compare these over time. This work in progress discusses our experience with open source tools on open source data and some of the challenges we encountered scaling our methods for big data.
Keywords
202
Thumbnail
1:16:05
226
242
FingerprintSet (mathematics)Open setGeomaticsFile archiverBasis <Mathematik>Computer animation
Personal digital assistantSet (mathematics)CASE <Informatik>
BuildingEntire functionData modelOpen setPoint (geometry)Characteristic polynomialSingle-precision floating-point formatPoint cloudDigital signalSurfaceMetadataPolygonLine (geometry)SoftwareOpen sourcePairwise comparisonTask (computing)SurfaceWebsiteResultantEndliche ModelltheorieProjective planePoint cloudTraffic reportingOpen sourcePolygonDifferent (Kate Ryan album)Square number1 (number)Raw image formatType theoryPopulation densityDocument management systemLine (geometry)Internet service providerTesselationSoftware testingCharacteristic polynomialPoint (geometry)Data compressionBitHistogramElectric power transmissionNoise (electronics)Pairwise comparisonComputer fileCuboidPhysical systemProcess (computing)PlotterAreaParameter (computer programming)File archiverMathematicsService (economics)Doubling the cubeMappingDigitizingAuthorizationComputer animation
Point (geometry)UsabilitySet (mathematics)ForestPatch (Unix)LaserHistogramDatabase normalizationSupport vector machineAutocovarianceVirtual machinePlanar graphPoint cloudAnisotropyScatteringSurfaceBounded variationVertical directionPlug-in (computing)Error messageSet (mathematics)BitPoint cloudLevel (video gaming)Point (geometry)Line (geometry)Virtual machineSupport vector machineAreaLatent heatPatch (Unix)Product (business)QuicksortForestHydraulic jumpBlock (periodic table)HistogramSquare numberDifferent (Kate Ryan album)MereologyPattern languageAutocovarianceComputer animationProgram flowchart
BuildingObject (grammar)Electric power transmissionObject (grammar)Network topologyBuildingPoint cloudIdeal (ethics)Type theoryComputer animation
Population densityArea
Database normalizationPoint (geometry)Numerical analysisPixelGroup actionBitPopulation densityLevel (video gaming)Computer animation
Radiology information systemDatabase normalizationType theoryPopulation densityComputer animation
Entire functionDiscrete element methodPoint (geometry)Point cloudProcess (computing)Graphics processing unitCoroutineScale (map)ImplementationBitProcess (computing)Computer fileProjective planeCoroutineServer (computing)Point cloudObservational studyScaling (geometry)CASE <Informatik>DataflowTensorComputer animation
Right angleSlide ruleObservational studyAuthorizationCASE <Informatik>AreaComputer animation
Advanced Encryption StandardHazard (2005 film)Natural numberDifferent (Kate Ryan album)AuthorizationMassGraph coloringRight angleProfil (magazine)Hill differential equationMappingOrthogonalityDigital photographyEvent horizonPort scannerComputer animation
Natural numberHazard (2005 film)MathematicsSet (mathematics)Event horizonDisplacement MappingComputer animation
Sweep line algorithmFlash memoryMetreMappingDenial-of-service attackComputer animation
Denial-of-service attackRemote Access ServiceAreaScale (map)Process (computing)Open sourceCovering spaceOpen setPersonal digital assistantVariable (mathematics)LaserHazard (2005 film)Natural numberMathematicsTexture mappingMappingEndliche ModelltheorieEvent horizonDenial-of-service attackStudent's t-testType theorySquare numberEntire functionMultiplicationImage resolutionOpen sourceProcess (computing)Extension (kinesiology)Point (geometry)ConsistencyCASE <Informatik>Time zoneScaling (geometry)Open setComputer animation
Transcript: English(auto-generated)
Hello, I'm so impressed so many people made it to the last talk So my name is Christopher Knuth I'm a geoscientist. I consider myself I'm interested in geomatics remote sensing and together with my colleague Sonia Greve We have been looking at the open light art archive in the Norwegian government shall I say and
I guess what inspired this talk for me was meeting the people developing open topography if anybody knows what that is Where you can access and so the basis of this talk is basically to showcase that data set I'm gonna go through a little history and then I'm gonna talk about the data and how you get it
I'm gonna discuss shortly some of the methods but very Not detailed and I'm gonna discuss some of the accuracy of the data sets that are on there And then I'm gonna talk briefly again Classification and issues we've had actually scaling which I've heard many good talks about so far and then end up with use cases
So in 2010 the Norwegian government passed the spatial data act basically what this did is this like paved the way for opening up a national infrastructure for housing and Actually distributing geospatial data. The other thing this did was that insured anything publicly funded Acquisitions would be given to the public to be able to download
so in 2016 the government Started a a project called the national detail terrain model project and what their mission was to create a one-meter DM over the entire country and basically using laser scanning All points or all ground coverage greater than two points per square meter
How are they're gonna accomplish this was using only private companies and then all data was made publicly available Now you can go onto these websites. There's two different Front ends or websites now and you can actually if you I think you have to register
But maybe not even not you can search the archive and you can just download laser data Currently as of last week, this was the coverage and The majority of the coverage is light are but there's areas where they only do photogrammetry because there's not much Vegetation so then they choose to only use photogrammetry and air flights on this
Status as today is 230,000 square kilometers or laser scanned I write 12 terabytes of compressed data, but this is like double compression So it's not just las it's then all the last files compressed again 13,000 individual projects which are individual acquisitions from one flight and
Then what they're doing is using private companies. So the private companies are Acquiring this stuff processing the data and submitting it back to the mapping authority Some of this work is maybe a year too old
So these are a bit outdated But it gives you an idea of like the ground point coverage over in the entire country and the most the highest acquisitions are usually where the populated places are and along corridors we have roads and rivers and Landslides and other things The other thing you'll notice is that there's a temporal Portion to this of course the project didn't start until 2016, but you see data from back in 2003 years now available
this is quite interesting because then you can start thinking about doing change detection you can start comparing and Looking at looking just at the histogram of the projects you can see there's quite a data from before 2016 here and then on this plot you can see anything that's darker has more temporal coverage
and so it's a majority where the people live Basic characteristics of the data is that all the data is chopped up into a national grid and it's actually a standard grid and they're about one to two kilometer tiles and
What the companies have to do they have to provide original point clouds and they have to be Classified some basics some are more advanced so the basic ones have ground noise and everything else Some of the more advanced ones have power lines and everything else so you can do any type of classifications and test them
The company's also had to provide digital terrain models digital service models You get metadata which includes flight lines border polygons point density and a project quality report which In my opinion is quite limited the reports of these don't really show exactly how they tie their data to the surface of the system
but you can check it out and So the methods that we used are mainly open source of Python PDAU or poodle or whatever
QGIS um we also looked a bit at cloud compare white box and last tools I'm not really showing much of this, but I'm just gonna show some results or talk about some results and One of the basic even though all these DMS are created already we decided let's do it ourselves So we take our raw point cloud through the Smurf filter And then we ended up getting ground classified and we get four products out of this a surface model a terrain model
height over ground model and a density We compared and we tried also the last tools the p-tin and the white box tools And I think the results we have is that the Smurf filter is the fastest most robust and you can actually
pre-parameterize it based on your topography and That makes it really automated if you know which parameters you want for the type of topography that you have so this is really awesome awesome job PDAU and Here's some of them comparisons. I just grabbed a region This is actually not far from my house and showing one area
that's got three acquisitions one in 2013 one in 2015 and one in 2017 and looking at the ground point coverage is exactly almost like you can see it on the map here on the histogram the 2013's is less than two points per square meter 2015 you're getting different lasers, and you're getting above two and then in 2017 you have a really great coverage with up to four almost ten
This is just a forest patch here because I talked a little bit about vegetation as well So this just shows you some of the data some of the qualities of the data now. I'm just going to compare What we did when we do to a DTM from these two products just to see what's the accuracy of?
the raw data that we're getting and so Generally the accuracy is quite good Plus minus 25 centimeters that's within the specifications in the Norwegian government when you look at the data You'll see the flight lines here in purple and so I guess this is something very common Is that I think it's problems of the the gyre and lack of maybe correction of it
So and some of these jumps will be up to about half a meter, but that's within the original data Maybe there's some corrections can be made Now choosing five or three other places here just a spot check We see the same pattern with these flight lines
Here is a valley and so up on the steeper parts of the valley you see that the errors get a little bit more and That just depending on where that light are actually reflects from how big the footprints are that sort of thing but up here Here's where you see a difference between the light are and then a photogrammetric DM
and so what you're seeing here is I think errors of the block adjustments and You see that in green down here that their accuracy is actually maybe about a meter or 50 centimeters So just some qualities of the data sets still phenomenal Once we get our ground out. We also have now then a non ground point cloud, and we're like okay
What methods can we use to begin segmenting this I've saw some great talks yesterday using machine learning We also tried, but we used cloud compare Because they have canoe po which is using support vector machines but before I did that or we looked at the dome and
Covariance features and We extracted all these in Python just using scipy simple clustering unsupervised and We can get out vegetation And just to show you a bit of an example there here is a colored point cloud
You have buildings trees roads and some other things and then when you pull out the ground you get everything else And then you throw it through a simple k-means it goes really fast in Python, and you get man-made objects I'm not going to go so much more in a detail here because that's a kind of ideal situation of course you have other
things power lines and stuff like this But still this type of stepwise approach down to classification segmentation is is what we want to do so we tried out also Canupa, and I'm going to go into some of the details there in a second because I had major problems scaling it But here's just a vegetative area and we wanted to calculate a vegetation density
But a vegetation density is very dependent on what kind of coverage you got what kind of laser, so how are you going to? compare So I normalize it and so When we count the number of points within pixels and normalize it by the ground coverage or an action off the ground covers a total
Coverage we can get like a density map, and you'll see like here. It's a little bit Darker here, it's more dense, and I think That this is actually related to topography as well steeper slope here So on these types of things you're not getting actually it's not really a vegetation density It's actually something that needs to be corrected
But this is what we're working towards Just to discuss a little bit We had major issues scaling when you're trying to run this on 13,000 project or 1300 projects which probably consists of about 20 I don't know 50,000 files I Have big server lots of RAM still took six months to process at least
so I Also with the point cloud classification unsupervised was really cool and fast, but we wanted something supervised but we couldn't scale up what was built in cloud compare and I don't know I started I've been seeing some great talks your talk yesterday Brad
And I'm wondering if I should I'm a little bit inexperienced in this so I wonder if I should be using Postgres Postgres and twine And also Developing routines I think for GPU processing tensorflow pytorch, and I think that would maybe speed up the issue any solutions available I saw some great talks yesterday, so hopefully soon
So just to show some case studies of this data, and what it's being used for so in 2020 right before to 2021 there was a quick clay slide, and this is not far from where we work in In Oslo, but an area with a whole bunch of houses suddenly gave way
And you see this car almost drove right into it The authorities have done quite a few scans, and so this is just to show you some of the You'll notice right up here. This is an ortho photo this is then a hill shade from the year before and
then after And so The data is quite accurate, so you're getting quite an amazing precision of detail in this I'm going to zoom in in a second, but what you mainly see is that the mass here Slid down and filled up inside these small valleys
You can then difference these And you get two profiles. This is a cross profile here, and you see about a 20 meter difference and a Longitudinal profile where you see actually the the mass flowing down This was a quite a tragic event. I can show you what the Mapping authorities put together because this is pretty amazing when they showed
this because you're actually seeing the color of the houses that fell down And so they were using this to try and locate people another example this is quite a funny event they call a piece of rock up here the man and
For a period of about three to four months we had slow TV in Norway Where they had a camera focused up at this rock because they were afraid it was going to move down, and it never happened It kept going and then what they use Lidar and found displacements this is actually going downhill this way and they found that the rock was separating here moving up here this is from Lidar data and
Eventually it actually did come down not as big as they wanted it to but then Lena Christensen and them could do a nice study and measure the change in the amount of rock coming down so for landslide purposes for monitoring from measuring these these data sets are absolutely crucial and
Then the last example I want to show flood risk and flood mapping Was a 2018 in October there was a lot of snow in the mountains And it got very warm very fast and the rivers weren't ready for this and this swamped an entire village shall I say and a
Master student took the Lidar data and started using Hecaros which is a hydraulic model for modeling the flung and he was able to reproduce quite well the flood extent so for for Preparing for these types of events this data is absolutely you couldn't you couldn't run this model without this Lidar data
And then just to summarize High resolution Lidar data is available over in the entire country of Norway It's open Download it use it try to train some of its classified zone. It's not so well classified The quality it's variable just like if anybody saw Howard stock the early data is very low
Quality but the later data sometimes you're getting up to 25 meters per 25 points per square meter So you can find some treasures in here. It's multiple use cases change detection vegetation modeling and monitoring risk mapping
The open source pipe Processing is amazing. I'm just having trouble scaling. How can we process all this data in a consistent matter? If anybody has any suggestions let me know and that's it. Thank you