Development of a graphical user interface to support the semi-automatic semantic segmentation of UAS-images
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 351 | |
Author | ||
License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/68903 (DOI) | |
Publisher | ||
Release Date | ||
Language | ||
Production Year | 2022 |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
| |
Keywords |
FOSS4G Firenze 2022339 / 351
1
7
13
22
25
31
33
36
39
41
43
44
46
52
53
55
58
59
60
76
80
93
98
104
108
127
128
133
135
141
142
143
150
151
168
173
176
178
190
196
200
201
202
204
211
219
225
226
236
242
251
258
263
270
284
285
292
00:00
EmailUser interfaceComputer-generated imageryEvent horizonMathematical analysisGraph (mathematics)Military operationOperator (mathematics)InformationArtificial intelligencePattern recognitionInformation extractionGraphical user interfaceObject-oriented programmingSpectrum (functional analysis)UsabilityDatabase normalizationSet (mathematics)User interfaceObject-oriented programmingGraph (mathematics)Database normalizationElectronic signatureSlide ruleArtificial neural networkMathematical analysisSimilarity (geometry)Goodness of fitOperator (mathematics)Interactive televisionMereologyInterpreter (computing)Event horizonPoint (geometry)Machine learningBilderkennungMachine visionGraphical user interfaceVector potentialStandard deviationArithmetic meanRaster graphicsPresentation of a groupUsabilityProof theoryProcedural programmingMultiplication signIdentical particlesComputer animation
06:07
InformationMultiplicationSpectrum (functional analysis)Integrated development environmentObject-oriented programmingComputer-generated imageryMusical ensembleGraphical user interfaceGraph (mathematics)Object-oriented programmingIntegrated development environmentObject-oriented programmingMusical ensembleProof theoryUser interfaceSlide ruleConfiguration spaceSpectrum (functional analysis)MultiplicationComputer animation
07:35
StatisticsGraphical user interfaceVisualization (computer graphics)BildsegmentierungObject-oriented programmingMusical ensembleComputer-generated imageryoutputOperator (mathematics)Asynchronous Transfer ModeObject-oriented programmingElectric currentAreaImplementationPixelDifferent (Kate Ryan album)Subject indexingMilitary operationGreen's functionPrice indexWorkstation <Musikinstrument>Instance (computer science)Maxima and minimaGraph (mathematics)CybersexHistogramThresholding (image processing)Similarity (geometry)Social classVarianceInformationPartition (number theory)Focus (optics)InformationPixelGraphical user interfaceCASE <Informatik>Graph (mathematics)Object-oriented programming1 (number)BitObject-oriented programmingSelectivity (electronic)Different (Kate Ryan album)Graph (mathematics)Operator (mathematics)Pointer (computer programming)SoftwareAreaSet (mathematics)Multiplication signUniverse (mathematics)Open setSlide ruleProgramming languageArithmetic meanAuditory maskingMaxima and minimaSource codeVisualization (computer graphics)Point (geometry)Presentation of a groupView (database)NumberIdentical particlesLimit (category theory)Insertion lossElectronic mailing listCharacteristic polynomialDifferenz <Mathematik>Standard deviationUser interfaceComputer programmingSubject indexingFreewareComputer animationSource code
15:29
Thresholding (image processing)CybersexHistogramMusical ensembleSimilarity (geometry)Social classVarianceInformationPartition (number theory)PixelFocus (optics)Computer-generated imageryGraphical user interfaceAsynchronous Transfer ModeGraph (mathematics)AreaWeightVertex (graph theory)Graph (mathematics)Maxima and minimaMereologyFunction (mathematics)Set (mathematics)Dedekind cutDatabase normalizationBildsegmentierungEigenvalues and eigenvectorsCondition numberMatrix (mathematics)Standard deviationElement (mathematics)DiagonalReduction of orderBerechnungskomplexitätOrientation (vector space)Term (mathematics)Characteristic polynomialGreatest elementSoftwareComputer configurationObject-oriented programmingSystem identificationPrice indexMenu (computing)Object-oriented programmingResultantProcedural programmingNeuroinformatikFilm editingCASE <Informatik>Position operatorSelectivity (electronic)BitPairwise comparisonGraph (mathematics)1 (number)Orientation (vector space)ComputerGraph (mathematics)Object-oriented programmingVector spaceDifferent (Kate Ryan album)Nichtlineares GleichungssystemObject-oriented programmingInformationSummierbarkeitLine (geometry)WeightUser interfaceDialectAuditory maskingAdditionPixelThresholding (image processing)Social classSimilarity (geometry)Arithmetic meanAsynchronous Transfer ModeGraph (mathematics)VarianceComputer-assisted translationDistribution (mathematics)MereologyPoint (geometry)HistogramMusical ensembleBimodal distributionStatistical dispersionConnectivity (graph theory)Normal (geometry)Eigenvalues and eigenvectorsMixed realityAreaDirection (geometry)Computer animation
23:24
Operator (mathematics)Graphical user interfaceObject-oriented programmingComputer-generated imagerySpectrum (functional analysis)MultiplicationGraph (mathematics)InformationGraph (mathematics)Computer animation
Transcript: English(auto-generated)
00:00
Thank you very much Maria, so During this presentation. We're going to talk about Work that we are the an interface actually a graphical interface that we are developing here in order to properly support the segmentation of Images in particular
00:21
This is motivated by As you probably know I mean during the last decades we had A lot of new instruments providing a huge amount of data Which clearly it's a very good thing because you go you can
00:40
definitely Obtain a much better knowledge about the reality you can try to Have good interpretation about events and so on however This also Leads you to some to some issues related to the analysis of this data, so
01:03
we do need the possibly some kind of automatic or semi-automatic tools that allow us to ease in some way our Understanding of these images our analysis of these images I guess that you may probably be already quite familiar with the idea of big data
01:23
And so this actually what we are talking about so we would like to Get some ways in order to ease our analysis of this big data So here we also have a question so too big or not too big that is the question
01:43
Okay, so likely speaking a lot of our Data are actually can actually be photos images, or if you prefer even a raster or whatever, but the point is that We do need
02:02
Since typically we do use artificial intelligence tools machine learning and deep learning tools in order to Analyze these images we do typically need quite large data sets in order to properly train these tools Unfortunately generating the data sets that we're going to use in order to
02:24
Train these tools. It's quite time-consuming this typically requires the presence of a lot of work by means of human operator and Despite vision some sense. It's quite a standard way to do it
02:40
It's still something that it's not really so so nice to do some typically quite annoying and so the idea here is that of supporting this human operator in such a way that The These operations versus these labeling is still going to be probably quite an annoying procedure, but
03:03
faster is possible and possibly and also more accurate if something Can also Enable some kind of Same automatic analysis of you now of the image itself
03:22
So We are quite lucky because during the last decades where we have a lot of Tools that have been developed in order to make image analysis and actually We are going to exploit
03:41
in general we would like to exploit the potential of also the new methods such as What we are going to talk about is trying to ease the generation of a ground truth data set in order to properly support
04:00
The learning part of these two obvious artificial intelligence based tools Okay, so Just to summarize what we are going to talk is trying to generate the ground truth data that we're trying to ease the generation of Ground truth data. So all of the labeling part and all of the segmentation labeling part part of the work
04:28
and since usually a lot of human interaction is required what we would like to have is something like Making it faster. And so the faster the better We are going to make some assumptions
04:40
So first we are going to assume that we are going to work with UAS images So this actually means that we're going to have a lot we typically are going to have a lot of redundancy between Our data because usually we acquire For example an image every second or something similar
05:01
So typically you have a lot of overlapping between these data And so also a lot of a lot of redundancy that we are going to exploit in In whatever we are going to show in the next slides Furthermore we are also assuming that we do have some multispectral information So we're going to deal with multispectral images and in particular
05:24
We're going to make to make an implicit assumption That is that the objects of our interest can be in some way Distinguished from from each other by means and in particular from whatever is not of our interest by means of their Spectral signature is not so surprising is something that is done quite in a standard way, but it's
05:46
It's quite important with respect to what we are going to see in the next slides This in some way the last assumption can also be summarized if you like as Everything you do is a signature of yourself or if you prefer everything I do
06:03
Especially during this presentation going to do it for you Okay So just to make the things clear how we are going to consider with the motivation the real motivation of this work Is that we collected a lot of images?
06:21
in order to make plastic object detection Flable environment so our original idea was that of properly dealing with this kind of images so We started developing this kind of interface in order to help ourselves To produce of the ground truth data for
06:42
properly training a machine learning tool for the classification of for the Plastic detection of object for the detection of plastic objects in this kind of problem So you as you can see in this slide actually we what we use is a multi spectral camera of the Maya camera
07:04
in the Sentinel to configuration So we use actually nine bands and I'm multispectral channels and we mounted this camera on the mattress of 300 and we collected something like some thousands of images of plastic objects in
07:21
In on the Arno river in this in the example that we are looking at in in this slide So the real motivation is first try to help ourselves in in making this thing So what we've what we have started to do is developing
07:40
This graphical user interface the graphical user interface that we're developing is in MATLAB Well, the first question in your mind is probably right now. The MATLAB is definitely not a free tool however most of well most of the of universities and in the academic institutions
08:04
We typically do have a MATLAB license. So the idea here is that of providing a Freely downloadable tool with whatever with all of the source code and whatever for a
08:20
Software which actually for a programming language, which actually is not a cannot be open in any way right now however, you you can still it's still free in some sense for academic use so That is the reason why well, obviously we also do it in this way because we are quite familiar with this kind of programming language
08:42
So that is the reason why we started doing it in this way So in the next slides we are going to focus on the free kind of operations that are typically that can typically be done by the by the guy so The first operation is just related to the visualization of the data
09:03
The second one is related to the segmentation and in the third case We are just going to talk about the object classification and in particular We're mostly going to focus on the second step. So on the segmentation step Well for what concerns
09:21
viewing the visualization of the imagery while this is quite standard so We can typically we provided the possibility of making quite standard operations such as Well, obviously selecting the data set navigating the data
09:41
Zooming shifting whatever out changing the visible channels and so on. So all of these are quite standard operations So I'm not going to spend too much time on this So instead for what concerns of the segmentation so we distinguish it between editing an already existing object and
10:02
Instead making a new in certain information about new object so okay, so in order to Consider this thing first. We should make an assumption. So here we assume that the objects can be distinguished by
10:21
by properly defining a mask on each of you each of our images and in particular It's quite important it's worth to notice that currently According to our assumptions if we have two overlapped objects, even if they are different from each other
10:41
We actually do not distinguish between the two of them This means that we do only have one mask for each images. And if we have two overlapping objects We are just going to consider them as Just one object. This is clearly a limitation We are also thinking about extending the finger extended this kind. So generalizing a little more
11:02
This kind of finger actually Generalizing a little bit more means in some sense Changing from a binary mask to an index of mask with different numbers and so on Depending on the object that we are considering but by the way for now, we are just making this kind of assumption
11:22
Okay, so we've also developed quite a lot of tools in order to support the Segmentation of the manual segmentation of the objects And also some semi-automatic tools that we're going to talk in a minute As you can see the list of the tools is
11:43
Relatively long. I mean not really incredibly long, but it's relatively long. We are just going to focus on The free for tools which are in some way More remarkable. So let's first start with the more obvious obvious ones
12:00
so in particular the free and selection here, there's nothing really so surprising so you can Just select with a standard the free and to the object each of the objects of your interest So I'm not going to provide too many details about it because it's pretty obvious and also
12:21
We supported the pixel that Each pixel selection. So in this case just by mouse clicking you you can Directly select or unselect the pixels of your interest This typically can be fought as an operation just to slightly modify an object that you have already defined
12:41
Okay, so it's pretty clear that the two tools that we have just talked about are quite obvious So let's move on something a bit more reasonable at least a bit more interesting So in particular we developed a couple a couple of tools for same automatic
13:02
Area selection of each of the objects. So in this case what we would like to obtain Something like a tool where the operator just select with We were left with a left click of the mouse any of the pixel of the object and then the tool should Automatically find whatever are the borders of the object. Okay, so this is the idea
13:26
So here you can see an example on how it works You should see that the in this case the user just selected. Do I have a pointer here? yeah, so you see that here the user just have just selected a point inside of the object and
13:41
The tool automatically selected the borders of the object. So in such a way to ease The labeling the segmentation labeling of the object. Okay, fine. So obvious thing can work. Well, we have Developed a couple of different ways in order to do it
14:01
but before describing the ways Let's just focus on a couple of assumptions. So we we assume that The user as we have already said just select a point inside of the object and the other assumption that we're making is the view of user also
14:21
Properly set as already probably set in some way The size of a searching window. Okay, so we do know that The objects should have a maximum size in a certain sense. Okay around the point that we have selected And also obviously for what concerns what we have already said
14:40
We also assume that in some way we can distinguish the object by means of its multispectral Characteristics from whatever is around it Okay, fine so We've implemented as I told you a couple of ways to do it the first it's just based on the also some also method for
15:01
segmenting actually in this case This pretty simple, I guess that most of you have already heard about it So let's try to describe it in a minute. So what we are going to do, let's focus just on using one channel You may imagine that there are many ways in order to combine the information from different channels
15:24
But for simplicity for the sake of simplicity in this particular in this presentation Let's just focus on the case in which we are just considering one channel So in this case, we are just going to Compute an histogram of Intensities inside of this image. Okay on the channel that we are considering in particular inside of a searching window and
15:46
What we are what we would like to find is in some way an optimal threshold for these intensities in such a way to distinguish between The Two kind of components that we are expecting so we are expecting to see two components one related to the objects
16:05
so you see here we have a multimodal distribution obviously Where one of the mode is related to the object and the other one is supposed to be related to the background in this Case we would like to distinguish between the two of them in order to define in some way an optimal
16:22
threshold to distinguish with two of them We are going to search for it in such a way to minimize the Intervariance between each of the two classes. Okay. So inside of each of the two classes We would like to minimize this intravariance Since we are considering just two classes visit equivalent in some way also to maximize the inter class
16:47
Variance, so that is probably not really so so important But the point is that in some way we would like to minimize if you prefer the dispersion around the mean of each of the two Of each of the two distribution that you are going to consider in this case
17:03
So this is pretty simple as you might have imagined. So This is just the first way that we have used. The other one is instead based Graph is based on graph based segmentation in this case We have just implemented the normalized cats the normalized cats. Sorry
17:24
Up so in this case We are just going to consider a graph where each of the pixel inside of a searching window is going to be a node of the graph, so let's try to See the graph so this clearly Just a simple graph in order to have an idea about what's happening and in particular
17:45
For each of the edges between the nodes we are going to define a weight This weight is related to the similarity of the values of a node. Okay So in particular what we can what is possible to show is that in order to make
18:02
This kind of computation work. This kind of weight is going to be Small the smaller is this way The more different are the two pixels that we are considering the two nodes that we are considering, okay Okay, fine, so what we would like to find is
18:23
Something like a cut. Okay, so you see in this red line is actually cutting The graph in two parts where we are distinguishing, you know, I mean the kind of problem that we are considering We are distinguishing between the object of our interest and the rest of the other searching window
18:40
So in some sense, let's talk about the rest as something like a background So a Way in order to define this candle to compute this kind of cut maybe for example that of Minimizing the ways that were the ways of the edges that we are cutting
19:00
Okay, so let's some for example all of you have the weights of all of the edges that we are cutting with this kind of Line and a first way in order to do it Maybe that let's say also quite a naive way in order to do it Is that of trying to minimize the sum of these weights? Okay This is quite reasonable. But unfortunately this trial this tends to in some way
19:26
favors The generation the generation of quite small regions, okay, which is not really our case So we would like to make something which is Which is not? Affected by this kind of issue So in order to make it and to make it in some sense to make it in a more fair way
19:46
So not depending on the area of the region that we are segmenting We're going to consider the normalized cut in the case of a normalized cut We are just considering this kind of functional to be minimized Okay, fine
20:02
The good thing in this kind of procedure is that actually can be shown you can find the details in this paper this kind of problem can be can be actually Reposed as the computation of eigenvalues of eigenvalues and they get vector of this kind
20:24
On this kind of a question Sorry in this kind of equation And so the only thing that we really do need of the importance the most important thing that we really do need to do Is computing the eigenvector obvious kind of equation in particular what we are really interested to it's just the eigenvector
20:43
Associated to the second smallest eigenvalue Okay, fine In addition to What I told you we have also developed another tool which actually tries to exploit the results from previous images because as I told you since we are acquiring images with
21:02
with a UAV or UAS we have that Successive images are quite similar to each other. Okay, and we would like in some way to exploit this kind of information so what we're going to do in this case is that of Assuming that the images are acquired
21:21
That the camera is actually oriented in an ideal orientation And we also assume that some navigation data are available from this kind of from these two kind of Information we can exploit this information in order to define a proper Searching window to find a previously find object in the next images
21:43
Okay, so it's not that the such a difficult thing you can see here that the objects are moving here You're directing looking at the masks Okay of the binary masks and what we are looking at is just you see that we have defined
22:00
searching windows and so on and in particular you can also imagine that we have You may also try to develop different tools in order to distinguish which ones Of the areas inside of the searching window are those Corresponding to the real object in our case. We developed we implemented
22:24
Some kind of comparison based actually on descriptors on feature descriptors and Here you can see what we found as the best Let's say Medium point the centroid actually of a new
22:43
Plastic positions in the new image. You see that now you are directly looking at Some some of the bands of a real image acquired by our drone Okay for both concerns with classification instead is something that we are still working about it. So classification for now
23:02
It's mostly yeah, I've finished it's just based on the manual selection object and Just the manual selection can be propagated also in the way that you have just seen by means of this kind of of two But for now, it's mostly manual. We're going to work a little bit more
23:20
in our future development on this step So this is just to conclude You have just seen the kind of tool that we have developed in order to support the segmentation of images And Clearly it's something that we are still developing So on the foreseen future work you see that there are still a couple of steps that we are planning to do
23:45
Hopefully quite soon. This is all Thank you