dsm2dtm: Generate DTM from DSM for free!
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 156 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68459 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Europe 2024 Tartu41 / 156
6
33
35
53
55
59
61
67
70
87
97
99
102
103
104
105
107
111
121
122
123
124
125
126
127
128
134
144
150
151
155
00:00
Streaming mediaFreewareElectric generatorLecture/Conference
00:16
Time domainComputational physicsSoftware developerDigital signalData modelStudent's t-testNetwork topologyObject (grammar)CalculationSurfaceTowerSoftwareFrequencyCASE <Informatik>Process (computing)Normal (geometry)Raster graphicsPixelBuildingEndliche ModelltheorieData structureMedical imagingUniformer RaumGraph coloringProper mapData miningGoodness of fitFamilyUniform resource locatorOnline helpMathematical optimizationComputer simulationBitProfil (magazine)View (database)Task (computing)Satellite2 (number)Arrow of timeTerm (mathematics)Dependent and independent variablesLine (geometry)MereologyRight anglePoint cloudBelegleserDomain nameSource codePresentation of a groupDifferent (Kate Ryan album)InterpolationRepresentation (politics)Computer animationMeeting/InterviewLecture/Conference
07:19
Moving averageServer (computing)Parameter (computer programming)Installation artVirtual realityWindowResultantRepository (publishing)JSON
08:35
CodeWorld Wide Web ConsortiumCodeFunctional (mathematics)Wrapper (data mining)ResultantComputer animationLecture/Conference
09:06
CodeComputer fileFile formatMoment (mathematics)Functional (mathematics)Translation (relic)CASE <Informatik>Library (computing)RadiusParameter (computer programming)CodeRaster graphicsComputer configurationComputer animation
10:17
PixeloutputBuildingEstimatorArithmetic progressionTesselationSoftware testingFeedbackOpen setGoodness of fitMultiplication signDiscrepancy theoryElectric generatorPoint (geometry)Repository (publishing)Object (grammar)Raster graphicsComputer fileBitRight angleWindowProcess (computing)Branch (computer science)Confidence intervalStability theoryMarginal distributionVideo gameRadiusInterpolationResultantGrass (card game)TriangleRectangleShape (magazine)Term (mathematics)Order (biology)AreaVirtual machine2 (number)Type theoryMultiplicationServer (computing)Image resolutionComputer animationLecture/Conference
16:59
Computer-assisted translationComputer animation
Transcript: English(auto-generated)
00:00
Hello everyone, I'm Naman and today we will be talking about DSM to DTM and like Mention how you can generate DTM from DSM for free It's publicly available. If you want to check it out. Just go ahead and check it out and let's get started Thanks for joining everyone Just a quick
00:22
agenda for the presentation We'll start with some introduction some brief introduction. What is DSM and DTM and some use cases? overview of this package a Demonstration and how you can use it and how can you contribute if you want to?
00:43
Okay, this I'm got a bit off guard because I was expecting some animation in this It's not there, but I'll just try to go ahead. So Rajat. He is a good friend of mine from uni He couldn't join. He helped me setting up with this package sometime back
01:02
He helped me packaging this package He helped me setting up get up actions. He has nothing to do with Geospatial he is in fact finishing his PhD in material science But he he did contribute in Some other ways so a big thanks to him I'm Naman. I am from India. I'm based in Berlin
01:23
I have some roughly five years of experience in geospatial domain. And what do I do? I'm looking for a job I Was impacted with layoffs a couple of months ago. And since then I at the moment, I'm actively looking for a job
01:42
Preferably based in Berlin, but open to anywhere if any of you is hiring or if you know anyone who is hiring Please hit me up. That would be of great help With that being said, let's get started So what is DSM and DTM? I believe most of you already know this I
02:01
Assume that's why you are here in the first place, but still that's of a brief introduction Both are raster data. In fact, both are a family of digital elevation models and Surprise surprise, they represent elevation data From Nadi or in other terms or simpler terms. I'm generally looking from top
02:21
Let's say as if you were sitting on top of the satellite or the sensor and you are looking down So that's that's that's like every pixel is representing elevation data from Nadi view DSM that's short for digital surface model It represents height of all the surface features above ground including human-made structures like building and vegetation
02:44
while on the other hand DTM which is short for digital terrain model. It represents height of bare earth surface without any human-made structures and Vegetation it can be derived from multi-reflect Lidar point cloud or it can be derived from DSM But today we will be focusing on how you can derive it from DSM
03:04
I like this image. This is a good visual representation of the difference between DSM and DTM on the left We are seeing an image So this green line if it was a DSM raster the green line each pixel would be representing the surface elevation values Which in this case involves?
03:21
Vegetation on the right hand side you are looking at a DTM So the brown line would be represented by the pixel values in a DTM raster just some use cases of DTM For calculating Heights using NDSM and DSM is a short for normalized Digital surface model basically you subtract DTM from a DSM and you get something called NDSM
03:45
And you can use it to calculate height of objects like trees buildings, etc For radio frequency network planning like some tower companies use some simulations to basically Try to find they try to find the optimum location way to put a cell phone tower to have maximum coverage
04:06
It is also used in engineering designs DTM's can be used to evaluate the suitability of a site for construction projects Such as highway highways pipelines and buildings and there are multiple more use cases. These are just few examples
04:21
Let's look at an actual example of DSM to DTM so what you are looking at right now on the right it's a student housing from my uni and This is the corresponding DSM Now I collected the data back in 2018 so it has been a while. I collected data using DJI phantom 3 advanced like multiple images overlapping images and the DSM was made using
04:45
Structure from motion. I'm using one of the propriety software's aegis of photo scan and this is the DSM It's missing the color ramp But the red values represent lowest elevation values and dark blue represents the highest elevation values
05:01
And this is the resulting DTM Let's take a closer look. Let's compare them. So generally speaking it gets the job done It had three main responsibilities like the first one was to remove man-made object or human-made object Which in this case was building it did remove that Second was to remove vegetation removing trees. It did remove that
05:22
There are some remaining vegetation It looks like I can't use it. Let me try again True Thanks So there are some trees over there
05:41
And the third task was to retain the terrain Profile which it did but let's take a closer look So if you look at this example this arrow this represents the tree which it did remove, but there are some shortcomings as well So this package does some post-processing which involves some interpolation and if you closely look at this part
06:03
Now ideally speaking we would have expected like a sharp distinction In this in this profile like but but as you can see like there is some interpolation and this these values Do have some lower elevation values which shouldn't have been there and this is because of interpolation
06:23
If you look at this part Now in the source DSM There so there was a bit apparently But again due to interpolation some of the values got interpolated and some of the values elevation values got reduced
06:40
Which shouldn't have been the case and This part it did remove the building But it did increase once again due to interpolation it did increase some of the elevation values, which is not ideal So it's not perfect, but depending on the use case it can get the job done So if your use case requires you to have an absolutely perfect DTM, I don't think this package is for you. I think
07:02
What might work for you is maybe working with LIDAR point cloud data and directly generating DTM from that But if your use case is somewhat simpler, let's just calculating height of buildings I think this might just work for you or let's say calculating trees heights and stuff like that. This might just work for you How to use it
07:22
By the way, this is also available on conda like you can simply install it using conda unfortunately, it's not available on pipe I But here I'll show some steps how you can use it on Linux using the repository itself. It's simple Just clone the repository
07:40
Make a virtual environment install install requirements and finally install saga and How you can run it simple just run Python DSM dot DC DSM to dtm.py and you pass the path of the DSM using this argument Dashtest DSM and it will generate DTM in a folder called generated result and it will also spit out the path of the
08:05
generated DTM and This is for Linux by the way for Mac It doesn't work. I tried it doesn't work because saga is not supported on Mac. I am personally using a Mac device I couldn't set it up on Mac So I would conclude it doesn't work on Mac unless some of you any one of you could set it up on Mac
08:25
Please let me know I Couldn't I have no idea about windows. I haven't tried it on windows If any of you try it on windows do let me know if it works or not. I have no idea Code walkthrough, so it extensively uses GDAL and saga throughout and we will be looking at some examples
08:43
So this function this is using sagas command-line interface saga does have an inbuilt functionality to generate DTM But the results are not perfect like so in a way I would say this package is more a wrapper around saga functionality, which involves a lot of post-processing and
09:00
Here you are looking at sagas command-line function to actually generate the DTM itself in the first place here you are looking at a GDAL functionality basically saga generates files in dot as that format I Explicitly tried to generate a file in dot JITA format as well it it didn't work, so I had to use some some other
09:24
Library to convert dot s dead file to jutif at least that was my use case I needed jutif so I used GDAL translate for that and A bunch of other functionalities as well like we will not cover all the functions, but just looking at some examples So this is one of the post-processing to close gaps once again sagas functionality
09:45
And this is to smoothen the raster once again sagas functionality You will see like there is a parameter Radius equals to two at the moment there is no option for the user to pass this parameter Unless the user actually goes inside the code and updates this value in the future. I do plan
10:03
Expose this functionality to the user so that the user can just manipulate these values as per their use case It will be there in future, but at the moment. It's not there unless you explicitly go inside the code and update this value Like mentioned before it's not perfect and for people who are concerned with the
10:26
Misalignment of this emoji, please stay with me this will all make sense Like mentioned I couldn't set it up on Mac. I wanted to set it up I couldn't so I ended up creating a virtual machine on GCP. I Still have it up and running if any of you who is using a Mac device who wants to give it a shot
10:43
Reach out to me. I'm happy to give you access to the VM as long as you don't exploit it Happy to do that It's slow So the example that we saw took like 30 to 40 seconds Depending on your input DSM. How big is that is and what's the resolution it can take up to minutes?
11:03
It's slow mainly because of saga, and it was it involves multiple steps But it's also slow because it generates a lot of temporary files It's not good engineering the last update was three years ago There are no type hints like it's lacking some of the basic stuff, but work is in progress like I
11:27
Wanted to have an example of tattoo like there is publicly available data. Thanks to this portal I wanted to have an example of this building where we could have run This package on this building and I did it did producer DTM, but every pixel had no data value
11:43
I don't know why probably because of CRS mismatch. I'm not sure I didn't get debug it But I'll be on that After this conference, I did start refactoring multiple times in the past I have multiple branches open in the package itself, but I think many of you can relate to this I started
12:03
but I did not finish so yeah, and That's why there's this emoji How can you contribute if you want to? Give feedback if you face any problems, please open up some issues If you see any improvements create a pull request Go through the to do's and if you find anything interesting, please contribute and there was supposed to be a screenshot of to do's here
12:28
Which is not there Where I listed a bunch of to do's which involve including adding tests Which actually is already done. I just need to merge the branch. I'm setting up by test
12:41
Setting up poetry adding pre-commit hooks adding documentation Moving the test files to a remote server or get LFS Removing all the temporary files generation and just passing them as a raster.io object and A bunch more once again, if you go to the repositories home page, you can actually look at the to do's
13:01
But yeah, if you find it interesting, please take a look. Give me feedback contribute if you can. That's all. Thank you Thanks for your time and happy to take any questions All right, so that was well well well within time but anyway
13:22
Before we started the questions, let's see who is using saga, please raise your hand Okay, we have two people so if I recall correctly saga was originally developed for Windows Right. Can you confirm that? Yeah, so do you use it on which system?
13:44
Linux here and you own windows. Okay, so at least those two and I'm a bit surprised it doesn't work on So yeah, you probably really want to get some feedback there Yeah before we go to the questions. I'm very curious about one thing
14:02
Did you consider doing this on grass instead of saga? I did not because Like I mentioned I did that like three four years back I did some Google search and saga was one of the top results and I just went with it Anyway, that's how life goes so who has questions then that was easy
14:30
Do you have any like margin or for stable for like confidence margin?
14:52
For the border of your DTM because since it seems to be an interpolation How do you manage to be? Sure of the elevation on the border of the process DTM. You are like a radius or something that can
15:08
Be considered as a margin like confidence margin so I Do not explicitly explicitly use any confidence margin to be honest. I don't know what that terms means But I have used this package on DSM switch were not of a regular shape that were basically not a rectangle
15:25
Maybe let's say a triangle So there was some area which was basically no real no data value and it does work on that like for the Margin, let's say for the borders. It just retains those original no data values Yeah, my question is more like you when you have like a DSM
15:42
Provided by tiles, you know, you have different tiles and so you have to process them Like in an order so in the end you have to match them, you know on the yeah, that's a good question No, I I did not try that I think Actually, that might be a good input point Because no I can see like because if you try on different tiles and again this package does some
16:06
Estimations depending on the input DSM like it calculates some parameter So if the input DSM is of different tiles, it can result in different DTM So on the border there might be some discrepancies in the value So I think that's a good input things and to answer your question. I did not okay
16:21
Yeah, actually that would be good yeah, just just open an issue thanks we have plenty of time
16:47
Don't you have a question ball? That's what the co-chair does. I mean if there are no questions, they can also just enjoy their evening It's a bit early to that. Okay? Yeah, sorry people. I tried