We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Laying the foundation for an artificial neural network for photogrammetric riverine bathymetry

00:00

Formal Metadata

Title
Laying the foundation for an artificial neural network for photogrammetric riverine bathymetry
Title of Series
Number of Parts
351
Author
Contributors
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language
Production Year2022

Content Metadata

Subject Area
Genre
Abstract
Laying the foundation for an artificial neural network for photogrammetric riverine bathymetry The submerged topography of rivers is a crucial variable in fluvial processes and hydrodynamics models. Fluvial bathymetry is traditionally realised through echo sounders embedded on vessels or total stations and GNSS receivers whether the surveyed riverbeds are small streams or dry. Besides being time-consuming and often spatially limited, traditional riverine bathymetry is strongly constrained by currents and deep waters. In such a scenario, remote sensing techniques have progressively complemented traditional bathymetry providing high-resolution information. To date, the peak of innovation for bathymetry has been reached with the use of optical sensors on uncrewed aerial vehicles (UAV) systems, along with green lidars (Vélez-Nicolás et al., 2021). The main obstacle in optical-derived bathymetry is the refraction of the light passing the atmosphere-water interface. The refraction distorts the photogrammetric scene reconstruction, causing in-water measures to be underestimated (i.e., shallower than reality). To correct these distortions, radiometric-based methods are frequently applied. They are focused on the spectral response of the means crossed by the light and are typically built on the theory that the total radiative energy reflected by the water column is function of the water depth (Makboul et al., 2017). The primary goal of the research on submerged topography is to understand the relationship between the water column reflectance and the water depth using statistical and trigonometrical models. The spread of artificial intelligence has given a new light of interest on spectral-based bathymetry by investigating the non-linear and very complex relationship between variables (Mandlburger et al., 2021). To train artificial intelligence models, large amounts of data are usually necessary; therefore, participatory approach and data sharing are required to build statistically-relevant datasets. In this scenario, FOSS tools and distributed resources are mandatory to manage the dataset and allow the replicability of the methodology. This work aims to test the effectiveness of artificial intelligence to correct water refraction in shallow inland water using very high-resolution images collected by Unmanned Aerial Vehicles (UAV) and processed through a total FOSS workflow. The tests focus on using synthetic information extracted from the visible component of the electromagnetic spectrum. An artificial neural network is created with the data from three different case studies placed in west-north Italy, and geologically and morphologically similar. The data for the analysis were collected in 2020. Each data collection was realised using a UAV commercial solution (DJI Phantom 4 Pro), and the following datasets were generated: i) RGB georeferenced orthomosaic of the riverbed and banks obtained from photogrammetric process, ii) georeferenced Digital Elevation Model (DEM) of the riverbed obtained from photogrammetric process, iii) GNSS measures of the riverbed and the riverbanks. The UAV-collected frames were elaborated through a standard structure from motion (SfM) procedure. Visual SfM was employed to align images and the 3D point cloud computation. The digital surface model (DSM) and the orthomosaic production were generated starting from the point cloud in Cloud Compare software. By applying the so-called direct-photogrammetry, the point clouds were directly georeferenced in the WGS84-UTM32 coordinate system thanks to the positioning information retrieved from the embedded GNSS dual-frequency receiver (Chiabrando, Lingua and Piras, 2013). Using the information regarding the camera position and the local height model provided by the national military Geographic Institute (IGM), the ellipsoidal heights were translated into orthometric heights. The GNSS measures had 3 cm accuracy on the vertical component and 1.5cm on the horizontal components. The RGB information, DSM and seven radiometric indices (i.e., Normalised Difference Turbidity Index; Red and Green Ratio; Red and Blue Ratio; Green and Red Ratio; Green and Blue Ratio; Blue and Red Ratio; Blue and Green Ratio) were calculated and stacked in an 11-bands raster (input raster). The Up component of the bathymetry cross-sections constituted the so-called "Z_GNSS" dataset and is the dependent variable of the regression. The position (Easting, Northing, Up) of each Z-GNSS observation was used to extract the pixel values of each band of the input photogrammetric dataset, including the photogrammetric DEM. The dataset was then normalised and divided into test (20% observations) and training (80% observations) datasets. In this work, a 5-layer multilayer perceptron (MLP) networks model with three hidden layers was built in Python using the deep learning library Keras with TensorFlow backend (Abadi et al., 2016). The ReLu activation function was added to the ANN layers to bring non-linear properties in the network. The dimension of the input layer is 11, and the weights are initialised to small Gaussian random values (kernel initialiser 'Normal') despite usually skewed or bimodal. A kernel regulizer, L1, was added to reduce the overfitting. The applied optimiser to update weights in the network is the Adaptive Moment Estimation (Adam) search technique, and the loss function, which evaluates the model used by the optimiser to navigate the weights, is the mean absolute error between the predicted output and the target output. The network was trained on the normalised dataset. The r-squared score, the Mean squared error and the Mean absolute error were computed. Finally, the permutation importance was measured using the eli5 python library. The neural network regressor performed over 0.80 of r-squared score on the test dataset. As expected, the permutation importance analysis reveals the high impact of the DEM and visible bands, and low importance scores are reported for ratios bands. The results are satisfying and quite relevant, although the model is the first step through a more complex and deeper neural network to correct water distortions in rivers. It has been trained on a relatively small dataset, but we intend to follow up with the research, add more data, and develop a free and open tool for the scientific community. The present work, provide a good insight about the high reliability and accuracy of artificial intelligence approaches in optical-derived bathymetry.
Keywords
202
Thumbnail
1:16:05
226
242
Artificial neural networkIntegrated development environmentArtificial neural networkResultantGreatest elementDistortion (mathematics)Goodness of fitComputer animation
Variable (mathematics)MeasurementMathematical modelMeasurementProcess (computing)BitWorkstation <Musikinstrument>Archaeological field surveyPoint (geometry)FrequencyDuality (mathematics)InformationGreen's functionRaster graphicsObservational studyFunction (mathematics)Medical imagingGreatest elementDoppler-EffektTotal S.A.Cross section (physics)Reflection (mathematics)Matrix (mathematics)Computer animation
RefractionRadiometryInterface (computing)OpticsSatelliteMeasurementFehlerschrankeData modelSurfaceAngleWater vaporAngleParameter (computer programming)Flow separationProcedural programmingPosition operatorSurfaceInformationDependent and independent variablesRefractionCausalitySpectrum (functional analysis)Different (Kate Ryan album)Reflection (mathematics)Interface (computing)Point (geometry)PhysicalismDigital photographyData structurePhysical systemArchaeological field surveyThetafunktionCartesian coordinate systemArithmetic meanType theoryGoodness of fitMultiplication signWordOpticsComputer animation
RefractionMachine learningArtificial intelligenceComplex analysisLinear mapSatelliteObservational studyArtificial neural networkVariable (mathematics)Endliche ModelltheorieArtificial neural networkData structureTerm (mathematics)Wave packetForestMachine learningMeasurementSoftware testingMultiplication signSound effectSlide ruleOrder (biology)Integrated development environmentGreatest elementRadiometryVirtual machineComputer animation
EstimationRadiometryRefractionData structureStatistical dispersionFocus (optics)FehlerschrankeArtificial neural networkMathematical modelMusical ensemblePhysical systemMultiplicationEndliche ModelltheorieWater vaporDuality (mathematics)Data structureTerm (mathematics)Mathematical analysisFrequencyArtificial neural networkDependent and independent variablesWave packetCASE <Informatik>Software testingObservational studyOrder (biology)Basis (linear algebra)Validity (statistics)Different (Kate Ryan album)Archaeological field surveyMereologyAreaoutputEntire functionWeightGreatest elementInformationProcedural programmingSurfaceSet (mathematics)String (computer science)PermutationStandard deviationMetreStreaming mediaComputer animation
TrianglePoint (geometry)Square numberSurfaceRWE DeaLine (geometry)Graphics tabletStreaming mediaNumberModal logicPoint (geometry)InformationState observerReal-time operating systemGreatest elementFrequencyDifferent (Kate Ryan album)Duality (mathematics)CuboidComputer animation
Image resolutionTelecommunicationModule (mathematics)Musical ensemblePhysical systemImage resolutionDirected graphMereologyObservational studyEndliche ModelltheoriePoint (geometry)AreaGreatest elementCASE <Informatik>Order (biology)InformationPhotographic mosaicWater vaporFehlerschrankeArmDigitizingMoment <Mathematik>Data acquisitionData structureComputer animation
Musical ensembleData modeloutputSheaf (mathematics)SurfaceAreaVariable (mathematics)Dependent and independent variablesAdaptive behaviorEstimationWeightMathematical optimizationFunction (mathematics)Linear mapInsertion lossPerceptronArtificial neural networkPerformance appraisalPredictionAlgorithmPermutationMathematical analysisStandard deviationReal numberSubsetFehlerschrankeAverageScale (map)Discrete groupSoftware bugGreatest elementHausdorff dimensionMathematical morphologyCharacteristic polynomialData structureObservational studyAdditionAnalog-to-digital converterMeasurementVirtual machineMathematical modelWater vaporFehlerschrankeInterface (computing)Linear regressionDecision tree learningWave packetArtificial neural networkDifferent (Kate Ryan album)Software testingVariable (mathematics)FunktionalanalysisCASE <Informatik>MeasurementState observerIntegrated development environmentProcedural programmingGreatest elementData structureAlgorithmBitRankingMusical ensembleScaling (geometry)FingerprintLibrary (computing)Mathematical optimizationPresentation of a groupType theoryMoment <Mathematik>Dependent and independent variablesGoodness of fitL-functionSimilarity (geometry)Set (mathematics)WeightInformationSurfaceResultantPairwise comparisonTerm (mathematics)Standard deviationMultiplication signDimensional analysisSquare numberTwitterNetwork topologyPower (physics)DataflowPoint (geometry)PixelRoundness (object)Line (geometry)Service (economics)Natural numberArithmetic meanArchaeological field surveyPermutationSlide ruleRight angleEndliche ModelltheorieMathematical analysisPole (complex analysis)Maxima and minimaPattern languageImage resolutionTensorMetreNormal (geometry)Direction (geometry)Stability theoryAsynchronous Transfer ModeGraph coloringInsertion lossAnalogyAbsolute valueComputer animation
Transcript: English(auto-generated)
Thank you, thank you, Paolo, for the nice introduction. Good morning, everyone, all few of us. I think it's because of the yesterday dinner. So I'm presenting the first results that my colleague, Vicente de Pietra, and I obtained for the creation of a simple neural network
for the correction of the distortion derived from the photogrammetric reconstruction of river bottoms. But let's try to contextualize a little bit this work. So bathymetry through the measure of the depth of waters
in land and ocean is a variable that is crucial and very important for the hydraulic modeling and for the study of river processing. So it can be measured with several methods. The traditional one, like topographic survey with total station and reflectors, but also dual frequency receiver,
like the one that we can see in the first image. We can use other methods, like profilometers, that are based on Doppler measurements. They are a little bit more faster than traditional methods. They work in cross-section along rivers,
exactly like topographic surveys. A little bit more expensive, but faster. Then we can use new technology, like green leaders. I didn't put it in the picture, but we can also have echo sounders that instead of working on cross-section, so on measure realized point by point,
they acquire information in a matrix way. So the final output is a raster to simplify it. So the real problem in the reconstruction of the bottom
from a photogrammetric application is connected to the refraction of the light in the atmosphere and in the water, because the light has a different reflection angle in these two means.
And the passage of the light within the interface between the air, the atmosphere, and the water cause a different, OK, yeah, I have a pointer, good, cause the structure for motion procedure to underestimate usually the point.
Because as you can see, theta 1 and theta 2 are different and defined by the means, so water and the air. As you can see here, this system can be solved quite easily if you know the two angles and the water surface position.
And what to solve this system, in photogrammetric world, several methods have been developed. We can call them photogrammetric or geometrical methods. They are well-performing, interesting, but unfortunately, they require a lot of time, a lot of information, starting from the parameters
of the camera, the position of the cameras, the water surface position, and quotes. So in time, other type of approach have been developed that are usually based on the spectral response of orthomosaic photos.
So instead of being in a physical system, we're now in, well, always a physical system, but it's based on optical information. So optical remote sensing have been developed different method and techniques
to correct the, to estimate survey the depth of the bottom covered by water. Such methods are usually developed for big rivers, lakes, and ocean.
Some of them rely on artificial intelligence and machine learning. Most of them are based on random forest and starting from early 2000, also method based on neural network are available. But most of them, as I said, are
created for oceans, sea, and lakes that have very, let's say, simple structure in terms of bottom respect to alpine rivers. So when we first time were in the situation of measure
and estimate the bathymetry of an environment like this one, which I, all the methods that I briefly recap you in the previous slides, and realized that we're not so well-performing. So Michael Licks and I asked ourselves, what can we do?
We try with traditional topography, but it was very time consuming. And so we thought that we can create an artificial neural network, a very small model that can be trained using data
that we already collected in all these alpine rivers in order to estimate the bathymetry. So what we did was test the effectiveness of empirical radiometric methods through deep learning
model to correct the underestimation of the bottom elevation derived from the structure for motion methodology. We, as I said, decided to focus only
on small water courses with less than one meter depth that are mainly busy in the alpine region of Piedmont in Northwest Italy. OK, here we have very short workflow. First, all the part of data collection
in which we realize a drone survey using a multi-rotor system, different sensor, different system. This doesn't matter in terms of analysis and quality of data for the model. Then we collect information of the depth
and also of the bottom of the river using dual frequency GNSS receivers and profilometers. The data set was then generated through a standard structure for motion procedure. And the depth was calculated as a difference between the water
surface, the Ws that I showed you before, and the real quote that were measured to GNSS and the profilometers. Then we prepare some extra feature
in order to improve the performances of the network. We selected different features that are basically derived from the radiometric response of our data. All data were scaled and divided into training data set,
test data set, and a part of it, an entire case study was used as unseen data set in order to check the ability of our net to generalize. We define the regressor network, perform a permutation analysis in order to check the importance of each the input
feature within the network, and then perform the validation on the test and the validation on the unseen data set. Our data were collected in three different areas. So they have in common of being alpine streams.
And they have pedals as the bottom with a granitic box. As you can see, we have a different number of observation. That's mainly because the data were collected with two different tools. So GNSS in our etaK modality, real time cinematic,
and the profilometers also embedded with dual frequency receivers. For the first two, we have 170 observation, while for Palice River, the last one, we have more than 600 observation,
which are basically points. So each point has x, y coordinates, so longitude and latitude, and innovation information regarding the bottom. Here you have the profilometers, as I said.
It has a dual frequency antenna embedded on it. And we are able to determine the elevation of the points in the bottom of the river, knowing the lever arm and the sinking of the vessel, too. As I said, we use two different cameras
for the data collection. Both are embedded on DJI drone system, our RGB cameras. We also have information regarding the infrared spectrum for two of the case studies. But for this specific work, we only focus on the visible part.
So very shortly, the errors and the accuracy that we got from the structure for motion reconstruction, we are around two centimeters and one centimeters and half of error on the reconstruction in the three areas,
and resolution of the oromosaic and the digital surfing model of one centimeters to four centimeters. Oh, this is the data collection with the profilometer. As you can see, as I said, it is fast, but not so fast, because not having an angina, and it's
to be moved from one bank to the other one really, really can wait in order to not move too much the water and create an error in the data acquisition. OK, so how we measure the depth.
We basically subtract the information of the water surface to the elevation of the points measured on the bottom. The water surface has been calculated in two different way
according to the tool that we use for the data collection. So for the GNSS with pole acquisition, we measure them manually along the interface between water and the riverbank, while for the profilometer, we used the quota measured by the GNSS antennae,
to which we subtracted the value of the lever arm, the sink, and the depth measure realized by the vessel itself. We then computed all the band rushes that you can see here on the right of the slide,
and scaled them using Schickett algorithm, the minimum access color. And all the data set from the very barrier, the one with 120 observation, was used as unseen data set or virgin data set.
Here there is the structure of the net. It's quite simple. It's a regressor, so we have the dependent variable, our depth, and the independent variable of the remaining including the depth with the BS caused
by the photogrammetric reconstruction. Everything was written in Python using Keras and TensorFlow as a back end. It is composed by a tree layer using two different activation functions. And the weight optimizer that we decide to use is the atom.
The loss function was measured on the mean absolute error, and everything was trained in 100 epochs. So as I said before, we computed also
an importance value for all the variables that we inserted in the net. The importance was calculated with a permutation analogy that basically considered the accuracy improvement or the crease for each of the variable
by training the model different time every time without a variable. The most important feature were the direction between red and blue and the NDTI that is the normalized stability index.
And then here I show you some results. The network performed quite well. The loss function converged very quick, as you can see. We have 068 on the test and 074 on the training.
This is mainly due, I think, to the bad generalization ability of the network for very swallow waters. So here you can see the direction between the real
depth and the predicted one. All the values are scaled. So in this case, we're not talking about centimeters or anything. As you can see, everything is under 0.5 as much more dispersed, while for deeper water,
the performance is much better. We can see similar trend also in the unseen data set. Indeed, the observation that you have here, observation 65 and around 90, are much more
in respect to the real data. So the trend is kind of well-predicted, but still there are some outlier. In this key, on the unseen data set, we have RMSE of 10 centimeters and an absolute error
of 8 centimeters. So these values are acceptable for the type of work that we are realizing on the one hand, because they are in line with RTK measures that
have crossing precision around 3 centimeters or less, and also because of the nature of the environment that we surveyed and we measured. As you can see, most of the patterns
that we have in the river bottom are more or less than centimeters. So obtaining procedure or accuracy, greater service mode in this have small meaning.
We see that the performance of the network is improved, it's better in high depth, but we believe that this is due to the dimension of the rock
in the bottom, the type of sediment, also the presence of vegetation. We believe that the performances of the net can be improved if we add more observation, because the demand gap is that we have a 500 observation for a neural network that even if it's a regressor,
they're still not enough. But this is only the beginning. We actually already add two more survey and two more data set to our net, and from the first results, we already see some improvements.
So for the future, we would like, of course, to enlarge our data set and try to investigate a little bit more on some aspects. So first of all, understanding why in the ranking of importance of the feature, the photogrammetric depth, so the depth of the water
estimated through structure for motion is not so important. We expected it to be linearly correlated to the real depth, so the corrected one.
This wasn't true, maybe because the permutation method is not the best one to check the importance of the feature between a regression, or maybe it's because of the scaling method, which is well-known being very, very influential
on the result of the model. Some others consideration on tests that we would like to do is the methodology for the measurement of the water surface, which we believe is still very critical, specifically for the GNSS measure relies on the pool,
because it depends on the user and the accuracy of the measure itself. So these are only the first step. We hope to be able to increase the net very soon and add much more observation that, for now, are
concentrated on the alpine arch. But who knows, maybe in the future, we can add also other type of environment, in fluvial environment. So thank you very much for the attention. If you have questions, I'm here.
Thank you. OK, so I see that there are no. So first of all, thank you for your presentation. Sorry, I'm a bit sleepy this morning, I believe, most of us. OK, so I see that there are no questions on venus-less,
but I don't know if the people here on the floor, they have questions or curiosity about this presentation. Oh, I see that there are. Oh, there is one person. You have to go there, where there is the microphone, please.
Yeah, I believe that it's working. It's already working. Ah, right now it is. Yes, can I hear you? OK, good morning. I'm also very sleepy. Pardon me. So yeah, Elin, a nice presentation. I really enjoyed it. I'm wondering something.
You have a relatively short set of observations, and you went for the neural networks. I'm wondering, did you consider regression trees? Because you can do very similar exercise with regression trees. And what are your thoughts about it, and why did you opt for this way? OK, thank you very much for this question.
Yes, we definitely consider also regressor trees. And actually, all the model that you see, all the data, were used to compare different methodology. One of them were the regressor tree, the standard linear regression with multi-parametric
regressions, and also artificial neural network. We decided to go for a neural network because of, well, the idea is to add much more data, and we believe that somehow is better able to generalize. Indeed, from the first result that we obtained
from the comparison, the regressor tree performed very well. We have a good score in terms of errors. But then when we apply it to an unseen data set, as we did with the artificial neural network, we have immediately a little bit more higher error. We're actually writing a paper for a journal about this.
So you will see the results soon. You're welcome. Thank you. Yes, and I see that there is a question also now, which is, you flow from 40 over 50 to 88 meter
above surface. What if you flew with a surface acting drawn and could be, for example, five meters above the water rock surface? Would that improve or worsen the precision?
OK, thank you for the interesting question. Well, we flew very, it's a relatively low flow as they say, 40, 80 meters. And this was because we needed a dimension of the pixel around one or two centimeters.
That is what was required. Of course, if we had to fly higher, we would have a larger pixel. So the spectral response of each of them would have been not so specific to each parallel or rock. And for sure, something would have changed.
So it would be interesting, actually, try to run the same test and develop the model also with the same data set, but maybe with a lower resolution in terms of pixel dimension. So that's it. I hope I answered.
OK, thank you. And is there another question? No, I don't know. So I believe at this point that we can close here. So thank you again for your presentation.