We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

An open-source-based workflow for DEM generation from Sentinel-1 for landslide volume estimation

00:00

Formal Metadata

Title
An open-source-based workflow for DEM generation from Sentinel-1 for landslide volume estimation
Title of Series
Number of Parts
351
Author
Contributors
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language
Production Year2022

Content Metadata

Subject Area
Genre
Abstract
An open-source-based workflow for DEM generation from Sentinel-1 for landslide volume estimation Digital elevation models (DEMs) are a representation of the topography of the Earth, stored as elevation values in regular raster grid cells. These data serve as basis for various geomorphological applications, for example, for landslide volume estimation. Access to timely, accurate and comprehensive information is crucial for landslide analysis, characterisation and for understanding (post-failure) behaviours. This information can subsequently be used to effectively assess and manage potential cascading hazards and risks, such as landslide dam outburst floods or debris flows. Freely available DEM data has been an important asset for landslide volume estimation. Earth observation (EO) techniques, such as DEM differencing, can be leveraged for volume estimation. However, their applicability is reduced by high costs for commercial DEM products, limited temporal and spatial coverage and resolution, or insufficient accuracy. Sentinel-1 synthetic aperture radar (SAR) data from the European Union's Earth observation programme Copernicus opens the opportunity to leverage free SAR data to generate on-demand multi-temporal topographic datasets. Sentinel-1 A & B data provide a new opportunity to tackle some of the problems related to data costs and spatio-temporal availability. Moreover, the European Space Agency (ESA) guarantees the continuity of the Sentinel-1 mission with the planned launch of another two satellites, i.e., Sentinel-1 C & D. Interferometric SAR (InSAR) approaches based on Sentinel-1 have often been used to detect surface deformation; however, few studies have addressed DEM generation (Braun, 2021). For example, Dabiri et al. (2020) tested Sentinel-1 for landslide volume estimation, but highlighted the need to further research and systematically assess the accuracy of the generated DEMs. InSAR analysis is often conducted using commercial software; however, a well-structured workflow based on free and open-source software (FOSS) increases the applicability and transferability of the DEM generation method. Although a general workflow for DEM generation from Sentinel-1 imagery based on InSAR has been described and documented (ASF DAAC, 2019; Braun, 2020, 2021), there is still a need for improvement, harmonisation and automation of the required steps based on open-source tools. Within the project SliDEM (Assessing the suitability of DEMs derived from Sentinel-1 for landslide volume estimation), we explore the potential of Sentinel-1 for the generation of multi-temporal DEMs for landslide assessment leveraging FOSS. Relying on the open-source Sentinel Application Platform (SNAP) developed by the ESA, the Statistical-Cost, Network-Flow Algorithm for Phase Unwrapping (SNAPHU) developed by Stanford University, and several other open-source software publicly available for geospatial and geomorphological applications, we work on a semi-automated and transferable workflow bundled in an open-source Python package that is currently under active development. The workflow uses available Python SNAP application programming interfaces (APIs), such as snappy and snapista. We distribute the SliDEM package within a Docker container, which allows its usage along with all its software dependencies in a structured and straightforward way, reducing usability problems related to software versioning and different operating systems. The final package will be released under an open-source license on a public GitHub repository. The package consists of different modules to 1) query Sentinel-1 image pairs based on perpendicular and temporal baseline thresholds that also match a given geographical and temporal extent; 2) download and archive suitable Sentinel-1 image pairs; 3) produce DEMs using InSAR techniques and perform necessary post-processing such as terrain correction and co-registration; 4) perform DEM differencing of pre- and post-event DEMs to quantify landslide volumes; and 5) assess the accuracy and validate the generated DEMs and volume estimates against reference data. The core module focusses on DEM generation from Sentinel-1 using InSAR techniques available in SNAP. The script co-registers and debursts Sentinel-1 image pairs before generating and filtering an interferogram. Phase unwrapping is performed using SNAPHU. The unwrapped phase is then converted into elevation values, which are finally geometrically corrected and co-registered to a reference DEM. Co-registration is based on assessing the normalised elevation biases over stable terrain (after Nuth and Kääb, 2011). We assess errors and uncertainties for each step and the quality of the Sentinel-1 derived DEMs using reference data and statistical approaches. The semi-automated workflow allows for the generation of DEMs in an iterative and structured manner, where a systematic evaluation of the resulting DEM quality can be performed by testing the influence of different temporal and perpendicular baselines, the usage of ascending and descending passes, distinct land use/land cover and topography, among other factors. Several major landslides in Austria and Norway have been selected to evaluate and validate the workflow in terms of reliability, performance, reproducibility, and transferability. The SliDEM workflow represents an important contribution to the field of natural hazard research by developing an open-source, low-cost, transferable, and semi-automated method for DEM generation and landslide volume estimation. From a practical perspective, disaster risk management can benefit from efficient methods that deliver added-value information. From a technical point of view, SliDEM tackles scientific questions on the validity of EO-based methods and the quality of results related to the assessment of geomorphological characteristics of landslides.
Keywords
Discrete element methodSource codeNeuroinformatikProjective planeComputer animation
Digital RevolutionInformation retrievalEndliche ModelltheorieGeometryOpen setVolume (thermodynamics)EstimationVolumeEvent horizonPerformance appraisalDiscrete element methodSoftware frameworkProduct (business)StatisticsCalculationQuery languagePairwise comparisonServer (computing)Computer-generated imageryDemosceneFile formatGroup actionDataflowCommon Language InfrastructureMiniDiscLocal ringScripting languageProcess (computing)Integrated development environmentComputing platformMultiplication signBitAngleSatellitePoint (geometry)Quantum stateTerm (mathematics)MathematicsDistanceMedical imagingServer (computing)Image registrationDifferent (Kate Ryan album)Volume (thermodynamics)Event horizonRouter (computing)Observational studyGoodness of fitInformationAreaValidity (statistics)EstimatorMessage passingPerformance appraisalSoftware frameworkPairwise comparisonTranslation (relic)Mathematical analysisEndliche ModelltheorieProjective planeUser interfaceHazard (2005 film)State of matterComputing platformImplementationTouch typingSurfaceControl flowComplete metric spaceWorkstation <Musikinstrument>Virtual machineParameter (computer programming)NeuroinformatikSeries (mathematics)Presentation of a groupUniverse (mathematics)Reference dataRepository (publishing)ResultantSoftwareGroup actionScripting languageMiniDiscDiscrete element methodElectric generatorDocument management systemVirtual realityComputer animation
BitParameter (computer programming)Document management systemSoftware testingAreaExtension (kinesiology)
Event horizonAreaBuildingSource codeObservational studyDigital photographyReference dataDifferent (Kate Ryan album)Document management systemEvent horizonAreaView (database)Square numberObservational studyWebsiteBitMetreMedical imagingContext awarenessBuildingComputer animation
Token ringSoftware bugNetwork topologyConditional-access moduleBit error rateQuery languageSystems engineeringState of matterMedical imagingCharacteristic polynomialOrbitAreaArithmetic meanMessage passingComputer animation
Computer-generated imageryScripting languageQuery languageFrequencyTemporal logicMaxima and minimaAverageTotal S.A.Medical imagingTemporal logicMatching (graph theory)Line (geometry)OrbitAverageMultiplication signElectronic mailing listMessage passingCASE <Informatik>Computer animation
Information securityFrictionMountain passWeb browserOpticsUniform resource locatorDiallyl disulfidePoint cloudComputer animation
Mountain passComputer-aided designReading (process)Order of magnitudeScripting languagePasswordLoginMedical imagingStapeldateiScripting languageBitServer (computing)Multiplication signComputer animation
Raster graphicsPhysical systemUser interfaceVisualization (computer graphics)Document management systemNumberComputer animation
Discrete element methodOrbitSubsetPixelScripting languagePrice indexThresholding (image processing)Parameter (computer programming)Query languageDemosceneComputer-generated imageryProcess (computing)Loop (music)40 (number)Range (statistics)Doppler-EffektStack (abstract data type)Polarization (waves)Parameter (computer programming)Scripting languageInformationError messageWave packetIntegrated development environmentComputer fileProcess (computing)Product (business)SubsetSoftwareNeuroinformatikFunction (mathematics)Discrete element methodComputer animation
Demo (music)Thresholding (image processing)AreaMatching (graph theory)Medical imagingThresholding (image processing)Computer animation
Scripting languageIntegrated development environmentCovering spaceOpen setDiscrete element methodCovering spaceAreaScripting languageMathematical analysisReference dataOpen setDiscrete element methodComputer animation
Scripting languageZeno of EleaEndliche ModelltheorieMathematical analysisDigital RevolutionModule (mathematics)MeasurementProcess (computing)VolumeStatisticsError messageRobuste StatistikInstallation artCross-correlationLinear regressionDiscrete element methodAbsolute valueIntegrated development environmentSample (statistics)Module (mathematics)BitRow (database)Discrete element methodComputer animation
Degree (graph theory)Discrete element methodNetwork topologyDemo (music)Error messageSample (statistics)AreaError messageDifferent (Kate Ryan album)Covering spaceFocus (optics)Normal (geometry)InformationCASE <Informatik>Network topologyAngleBitComputer animation
Derivation (linguistics)Plot (narrative)Pairwise comparisonDerivation (linguistics)BitDiscrete element methodComputer animation
Discrete element methodImage registrationDemo (music)Error messageAverageArithmetic meanSquare numberRootkitMedianAbsolute valueDatabase normalizationStandard deviationMaxima and minimaAreaEvent horizonVolume (thermodynamics)VolumeCalculationInsertion lossValidity (statistics)Source codeObservational studyCharacteristic polynomialEstimationDISMAInformationImplementationUsabilityoutputParameter (computer programming)Image registrationStatisticsGoodness of fitVolume (thermodynamics)UsabilityMathematical analysisValidity (statistics)HypothesisObservational studyElectric generatorImplementationAreaEstimatorSource codeDigital rights managementoutputParameter (computer programming)TimestampEvent horizonSoftware testingIdentifiabilityHeat transferDifferent (Kate Ryan album)EstimationDocument management systemNumberSoftware bugDiscrete element methodRepository (publishing)Computer animation
Hazard (2005 film)outputRepository (publishing)Computer animation
Transcript: English(auto-generated)
Welcome, everyone. Thank you for being here after this wonderful gala dinner we just had, so very brave of you. My name is Lorena Abad. I'm from the Department of Geopharmatics at the University of Salzburg, and I'm here to present to you some work of one of our projects called Sly DEM, and I'm here on behalf also of my co-authors, Daniel Hedling,
Sara Davidi, and Benjamin Rapson from the University of Norway. So, let's get started quickly with a bit of background, just to have a sense. Do any of you work with SAR data, so radar data? Yeah, some? Okay, so I will just give a little intro of the very basics
for myself as well, because I'm not usually working with SAR data. So, the idea here is radar is penetrating the Earth and getting information every time ray goes into the Earth and bounces back. The idea here is that if we have two satellites,
one in time one and one in time two, orbiting the air, we can get some information of the topography of the Earth's surface, because we have two points in time with an angle that will give us this information. Then, with this information, we have these two satellites,
they go at different times, we have a perpendicular baseline, which is something that I'm gonna just be throwing the term in here and there, and we have the temporal baseline, which is the difference in time, time one, time two, and the perpendicular baseline is the distance between the two satellites. The idea here is that because we're having these satellites orbiting the
Earth, we can get information about the topography of the Earth. If there is no change between two points in time when the satellites are orbiting, it means that we can get some information about the topography, the real topography of the Earth, that means the elevation
of the Earth. If there is any change between point one and point two, then we're gonna get information about the changes that are happening. An example of this is, for example, when you have volcanic activity, this is very used to check, okay, how much difference has it been from the state of before volcanic activity and after volcanic activity, how much
has it moved, or earthquakes, etc. But actually, for our purpose of doing digital elevation models and to get topographic data, this is the situation that we don't want to get to. We want to get to a situation where point one and point two are stable enough so that we can get the information
of the elevation and topography of the Earth. And therefore, we want that this time one and time two is as small as possible, that we get these two passes really close in time so that the information is not changing so much. So I hope this was a quick intro to SAR data, and therefore I'm going to show you what we want to do with it. So the project itself is
simple. We want to start with a landslide event because we're working with landslides at the risk hazard and climate lab, and what we want to know is, okay, from this landslide event, can I get digital elevation models from these two passes of Sentinel-1? So for that,
the idea is, okay, we collect the data, we check for their suitability, we generate a DEM with this information, we compute the landslide volume estimation because we can get a DEM before the event, we can get a DEM after the event. In parallel, we will do some DEM accuracy
assessment to see how good it is, and we will do some value estimation validation to see how good this is. So the idea is simple, and what we wanted to do is do a conceptual framework and a workflow that would allow us to do this systematic analysis of these elevation models that we're generating. For that, we chose several events and study areas for the workflow evaluation.
So we have a landslide in Kraknesalta in Norway, probably you saw in the news about it, some houses went away, it's really in the north. We have a quickly landslide in Gjerdum, Norway, which was in Christmas 2020, which was also quite fatal. And we have a rockfall in
Switzerland, Austria, which did not have any fatalities, but it is quite big that we thought this could be a good event and study area for the workflow evaluation. So this is the idea that we had, okay, we want to do this, but now how do we technically implement this? So don't
get overwhelmed, this is just a very simplified workflow of what happened, but I will show you more examples. But very simply, it's just a translation of what I showed previously. So you have Sentinel-1 image pairs, we want to query them, we want to get them, these two image pairs
that are closing time, right? Then we want to download these selected pairs, we want to generate DMs out of it, and for that we use Snap. If you work with Sentinel-1, you're probably very familiar with Snap. We want to encourage Easter and assess their quality, because if you've ever tried to do a DEM with Sentinel-1 data, you will know that the quality is not the
best. So the idea here is to get it at least to a decent level so that we can do baseline comparisons, and then we want to estimate the volume change. So here I've just put some of the key places where we, the key players of the workflow, so we query from the Alaska facility server, we use Snap, and here we use some information also from OpenTopography to get
reference data to do our coregistration and assess the quality, and then the volume change will be a difference, a DEM of difference between the two points in time. So what I did here is implement everything with Python, so with the Snap API for Python, and the idea is that
you can clone this repository and pull this Docker image to be able to run this workflow. Now why did I use Docker? In this case, if you've ever worked with Snap, let's go again for that,
you will know that it is a little bit of a mess to have all the dependencies on point, to use it between platforms, Linux, Windows, it's a complete mess, you will need to do some updates, some many things that have a lot of hassle in doing the implementation.
So I actually have my co-worker who is normally working with SAR data, and I remember her spending one or two weeks just setting up the workflow in a workstation to be able to process the SAR imagery. So then I thought, okay, this is not good, we need to Dockerize this, we
need to put this, everything in one single compartment that will allow us to just run everything in one single place, we don't need to be thinking about all this software. So this is why Dockerize it. How did I do it? It's based on the Mundialis Snap image, it's currently running with Snap 8 because Snap 9 just was released, so I didn't want to touch anything or break anything before
this presentation, but I promise I will update it soon. I compiled it via GitHub Actions because I discovered that our university server had some problems with doing updates from Snap, so I couldn't compile all this Docker image in there, so this is also something that you can
get around, so just call it from Docker Hub, and the way we're working is mounting a local volume to write our results, that means we can connect to our own computer to write the results to disk and nothing stays in the Docker or the virtual machine, if that's a
better way to understand what a Docker container is. And I wrote a series of scripts that you can pass arguments to in the command line to be able to run this. So now let's go with a practical example so that this gets a little bit less abstract and more applied, and
this is one of the landslides that I was talking about, this is in Gedrum, Norway, as you see many houses went out, it's quite big in extent, so we thought it would be an interesting place to look into. The topography is not as steep, it's a bit hilly, but not high mountains,
but we have also other test areas where there are high mountains, where there's more vegetation, just to see what exactly is going on and what are the best parameters to generate these DMs. So just a bit more context, the event was in the 30th of December 2020, there were some minor events after that, 10 people lost their lives, several buildings
were destroyed, several people evacuated, and then more to the data itself, it had a detachment area of 0.12 square kilometers, an outlet area of 0.26 square kilometers, 1.35 million cubic meters were mobilized, and almost 1 million cubic meters was also deposited. The study area is here in
Gedrum, Norway, closer to Oslo, and this is one of the images that my co-worker got when he went to the site to take a look at what was happening. This is a bit of an aerial view of what was before and after, luckily because this was a big event we have a lot of data where
we can check what was going on, a lot of UAV imagery, so like reference data to see how good our DMs are getting, and we have here the elevation difference, so what was detached, what went out of the landslide, and here the deposition, what was deposited after the event.
So let's go step by step on how this workflow will work. Step one before having the workflow will be to query data. If you've ever queried data from the Alaska facility, you will be familiar with these screens, so first what you do is draw an AOI, and then you get a
bunch of images in this area here that correspond to the area that you're looking into. Once you're done with that, this can also be by temporal search, just narrow down, once you're done with that, you will have to do a baseline search, meaning that for every image that you
found, you will need to find a pair that is compatible with the image that you selected, so that you can do this double pass with the imagery, so having a small temporal baseline, a good perpendicular baseline, and that they are both in the same orbit, that they're both in
the same orbit. This is how you would do it normally, and this is what we did to improve the workflow. In what way? If you would do this manually, you would have, for example, for this example here, we have the AOI, we have two months, so we start in July, 1st of July, and we end on the 31st
of August, and if you do this manually, you will get 59 geographic matches, so you would have 59 images. For each image, you will have an average of 355 images that you have to look into if they match or don't match. That means you have to look into 20,986 matches, and then from there,
you would have to filter how many you really want. In this case, it's 86. So with this first step, I simplified this in a way that this will all be done in the background because I created the Alaska API, where you will just get a list of your reference ID, your match ID,
the orbit, the path, and a temporal baseline, perpendicular baseline, reference dates, everything that you need. You don't need to spend all this time querying for the data. That's the bottom line. That's what also my coworker was taking weeks to do. Now she's
very happy. She can just run it once, get the CSV, and see what's going on. What I added also is a URL to the browser of Sentinel Hub so that you can take a look at how it looked exactly around these dates in optical imagery so that you see, okay, actually there was snow on these dates, so it's not so useful for me to do interferometric SAR because there will be too
much interference with my final workflow, or there was a lot of clouds. Maybe it was raining too much. Maybe the atmospheric is not going to be the optimal to do my interferometric SAR, so you can also choose here. From here, you can take a look at how everything is going. Then you download, and that's very simple. You just have the CSV, and I put here a column
where you, as an analyst, will have to change from false to true. And with that, you just pass this to the next script saying, okay, everything that is true, please download it. It will take time. Sadly, we have to download the whole image because otherwise
SNAP will not be able to process this. It is a bit of a quirk, but okay, that's what we need to deal with, and if you have a server, this will go much better in the batch download. And step number three is generating the DMs. This is how, let's call it, you would normally
do it. This is the interface of SNAP. It's, yeah, I'm not so used to it. I don't really like it. I don't know how to do all the visualizations and check everything, but this somehow it looks like. And just, I will not go through all of this, but this is every single step that you need to do to get a DEM out of it. So it's quite long. It's divided in three
pipelines because these cannot be done in one go, but then you need to get out of this SNAP environment to another software to do this very specific phase unwrapping. And then you need to back to SNAP and finish your last steps, which will be doing this DEM, terrain correction,
et cetera. So the idea here is that everything was bundled into one script. So the next one that will go, you can give a lot of parameters that you would normally give to SNAP. You can give them to the script and so that you don't get lost into, okay, what did I set
here? What I didn't set here? I created a log file where you will have all the information of, okay, what did I set? What were the parameters that were computed inside? So like the subset, the burs, the swats. You have a lot of parameters for each pipeline, and we have some information that is also computed during the process. As you see here
as well, you have some outputs already. Just to give you an idea, my coworker would spend like at least three days generating a DEM. Now this is done in half an hour, so it's quite useful. And we also have these intermediate steps because it's very important when you're doing a DEM to
look at the intermediate products and see, okay, is this really working well or not? Is the interferogram making any sense or not? Do we have any errors or not? So this is how the products look like. We get elevation, we get the coherence, which is how matching are the two images just to
see how good the elevation values would be on each of the areas. So higher areas means there is a better match between the two images. As you can see, this is not a great match, but this is one of the examples that I chose during this. And you can also have an elevation
filter by a coherence threshold. If you think this coherence is just too low, you can just get rid of this. And we have intensity, the phase, and the unwrapped phase. Finally, thank you, we have step four, which is assessing the quality. And the idea here,
I added another little script where you can get some reference data so that this is where the open topography enters. You can download the NASA DEM or any DEM that is available for your area. And also there is a world cover where you can do some analysis by land use, land cover, and to see, okay, where is my DEM elevation performing better or not?
And I use for this, this very good package called XDEM. So if you have also working with DEM data, I recommend you looking into it. I would recommend you do your own quality assessment using this package, but because we're trying to do this in several steps in an
organized way where we just generate several DEMs and assess them all in a row, that's why there's a module for doing quality assessment. And there's a little bit of a hint of what you can get out of it. So for example, we have the reference DEM, in this case is the
patchy. It's not super beautiful, but okay, this is one of the things that we get. And what we do is get an elevation difference to see, okay, where are those values performing better or not? So as you see, this area is quite bad, but the area that we're really interested in is the landslides. So this is what we focus on. It's still bad, but not as
bad as the other areas around. And as you see, you can get some information about how the qualities, the error, how it's changing according to the slope. So higher slopes have higher errors as normal because then the angles are harder to compute. We also have some
information about how it varies with the aspect. And because we included some land use, land cover data, we can also see, okay, with grassland and cropland, apparently it's doing okay, but with tree cover and sharp land, we have a bit more problems. And just to get a
general idea, I've also included these terrain derivatives where you can have a quick look of how it would look like if you're really working with this DEM. So just for comparison here, this is the NASA DEM, so a bit nicer. This is the Sentinel-1 DEM, very noisy, but still you can get a general idea of what you're working with.
And of course, the vertical co-registration, as I said. If we have a reference DEM, we can see, okay, how is this looking? Is this looking good or not? Some statistics that you can get as well to see, okay, how is the error good or not? And finally, calculating volume. So if you have a pre-event and a post-event, you can see, okay, what was the volume that was changing
between these two timestamps? And you have the elevation difference and some of the numbers that you get that here we can compute. This is really an overestimation of this particular event, but alas, this is also meant to do a systematic analysis of all the DEMs that we want to compute
for this area. So as a conclusion, we aim to develop an open-source, low-cost, transferable, semi-automated method, because you need to set which one to download, for DEM generation and volume estimation. We looked into several study areas, the test implementation, identify
input parameters, detect bugs, improve performance, assess the usability, how good this is really for long-term. We think that disaster risk management can benefit greatly from this, because you just don't have all the technical hassle, but really can focus on the analysis. And we think it is really useful to also tackle some scientific questions regarding the validity
of your data for all this geomorphological analysis. So we hope that this is useful, and yes, this is it for my presentation. Please feel free to go to the GitHub repository and take a look at what we've done. I'm working on the documentation, so if you have any
input on that, please feel welcome. Thank you very much.