Status of OTBTF, the Orfeo ToolBox extension for deep learning
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 351 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/69112 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Firenze 2022132 / 351
1
7
13
22
25
31
33
36
39
41
43
44
46
52
53
55
58
59
60
76
80
93
98
104
108
127
128
133
135
141
142
143
150
151
168
173
176
178
190
196
200
201
202
204
211
219
225
226
236
242
251
258
263
270
284
285
292
00:00
Extension (kinesiology)Integrated development environmentSource codeWebsiteGraph (mathematics)Computational physicsProjective planeNeuroinformatikCartesian coordinate systemEndliche ModelltheorieMathematical modelSoftware frameworkReal numberFunction (mathematics)CodeInstance (computer science)Slide ruleGraph (mathematics)Computer programmingOperator (mathematics)Set (mathematics)MultilaterationAdditionFile formatPresentation of a groupCuboidSoftware developerSoftware testingLevel (video gaming)Process (computing)Arithmetic meanWave packetComplex (psychology)Semiconductor memoryVideo gameBitData streamCovering spaceTensorOrder (biology)Graph coloringLine (geometry)Product (business)InferenceDot productRevision controlMoment (mathematics)Social classVirtual machineResultantExtension (kinesiology)Numerical analysisMultiplication signRow (database)CalculationField of setsArray data structureoutputFluid staticsMathematical analysisFunctional (mathematics)Goodness of fitGraphics processing unitTerm (mathematics)Sinc functionDistribution (mathematics)Flow separationData centerCompilation albumDataflowDifferential equationGeneric programmingIntegrated development environmentXMLComputer animationDiagram
09:17
Data modelScale (map)Computational scienceSupercomputerFile formatReading (process)Directory serviceComputer filePoint cloudMathematical optimizationCluster samplingMathematical modelInferenceImplementationUsabilityPatch (Unix)Social classDeep WebFunction (mathematics)SoftwareFile formatGraphics processing unitGradientObject (grammar)Row (database)Computer clusterMathematical modelInstance (computer science)MereologyCovering spaceForestPoint cloudEndliche ModelltheorieSupercomputerSoftware frameworkNormal (geometry)Operator (mathematics)Dimensional analysisFunctional (mathematics)Product (business)Point (geometry)Wave packetoutputExpert systemChainNeuroinformatikComputer fileCartesian coordinate systemComplex (psychology)ConvolutionInferenceReal numberDifferent (Kate Ryan album)TensorTesselationMultiplication signSoftware testingSemiconductor memoryMetric systemDigital photographyMathematical optimizationScaling (geometry)CASE <Informatik>Entire functionPolygonStatisticsMetreBuildingDialectFile systemImage resolutionCloud computingCuboidSource codeLevel (video gaming)AdditionVector spacePresentation of a groupDefault (computer science)Set (mathematics)Square numberTime seriesProcess (computing)Dot productWhiteboardGene clusterWeightFlagTranscodierungDataflowComputer animationXML
18:35
outputComputer-generated imageryComputer networkMathematical modelProduct (business)Texture mappingForestForestComputer animation
18:58
Image resolutionInheritance (object-oriented programming)Data modelMusical ensembleInferenceComputer-generated imageryFunction (mathematics)Source codeFree variables and bound variablesLogical constantTesselationCodierung <Programmierung>Point cloudOpticsoutputImage resolutionRight angleProcess (computing)Endliche ModelltheorieBefehlsprozessorMathematical modelPoint cloudProjective planeMultiplication signTemporal logicOpticsComputer animation
20:40
Computer-generated imageryRight angleArtificial neural networkoutputSeries (mathematics)OpticsAreaTime seriesDemosceneTemporal logicResultantAreaEndliche ModelltheorieComputer animation
20:59
Endliche ModelltheorieUniform resource locatorAreaObservational study
21:18
Reading (process)Pairwise comparisonComputer-generated imageryOpticsCNNMultiplicationTemporal logicFingerprintBlogPoint cloudDigital photographySource codeProduct (business)ApproximationData modelInversion (music)Artificial neural networkMoistureProjective planeMultiplication signRight angleCartesian coordinate systemBlogBitReduction of orderSemiconductor memoryMereologyInstance (computer science)Endliche ModelltheorieMappingSet (mathematics)Social classTensorDataflowMathematical modelSoftwareData centerConvolutionSupport vector machineInteractive televisionSoftware frameworkPatch (Unix)Selectivity (electronic)INTEGRALWechselseitige InformationBuildingWave packetConnectivity (graph theory)Field (computer science)Texture mappingProduct (business)ForestSlide ruleLimit (category theory)Compilation albumSampling (statistics)Artificial neural networkWebsiteComputer animationXML
Transcript: English(auto-generated)
00:00
Hi, I'm Rami Kresson, I'm with the French Institute for Agriculture, Food and Environment and I'll try to present you the OTPTF extension for deep learning. I knew it. Where's the mouse? Okay.
00:21
I'm sorry, I made my presentation with HTML. It's really difficult to move the mouse because it is very very slow and with some latency, you know. So in short, OTPTF is an extension for the OTPTF box
00:42
and it aims to provide the developers or user a generic framework to build deep learning based applications. I will detail what you can do with it in a few moments. Its main purpose was to do some research, to conduct some research like land cover mapping and image restoration.
01:05
But also for education and finally to put some research results in production. For instance in the Teilhard data centre. And yes, its goal is basically to use deep learning techniques so to do that you have to create some datasets
01:22
to train models and to apply them on real world remote sensing images. So you can apply model in OTP applications and this has several benefits. For instance, the data stream is continuous and from the images that you read to the images that you write
01:43
you need a very small memory footprint and the pipeline guarantees good performance in terms of computational stuff. Yes, okay. So it is a project which has started four years ago.
02:03
It has a lot of improvements since and I think the main improvements over the years are following the documentation of course. We provide some Docker images, Docker builds because compiling TensorFlow is a pain and compiling OTP is okay thanks to the super build
02:22
but compiling TensorFlow is like four hours of compiling stuff with BIDL so it's really difficult. So the Docker build is all that and we provide one image for the GPU using the NVIDIA Docker support for Ubuntu.
02:42
We also improved the CI and CD with static analysis and test, functional test, application test and finally some automated ship of the Docker images and soon the documentation will be shipped as well. And recently we had the TensorFlow 2 support.
03:02
Why it's so important? Because TensorFlow 1 was really ugly to use and for developers to debug it was really ugly. That's why a lot of people went to PyTorch but TensorFlow 2 is all that. It's really easy to code now and it has a lot of advantages
03:21
especially for distributed training. So I'll talk a bit about distributed training later. Very recently like two months ago we have released the last version with some Python classes and we can now talk about Python API. It's a set of classes that use the deep learning practitioners
03:42
to do their work. I will detail a bit of all that. You can clone the code here on GitHub and you can pull the Docker images on the Docker Hub. Okay, now it's the other row.
04:13
You know what, I think I will use that to go from left to right and the mouse to go down.
04:20
Yes, okay. So what's OTBTF? What's deep learning? What's computational graph? Here you have the fair toolbox which targets remote sensing images processing. TensorFlow is like a big calculator.
04:40
It works on computational graph which process multi-dimensional data arrays and using that you can do some machine learning, deep learning. So you can perform deep learning with OTBTF but you can also perform just numeric computations
05:00
for instance complex algebra stuff that you can parallelize on GPUs but you don't want to put the hand in the code low level. So you can just write some simple Python code using TensorFlow and shape the model into OTBTF
05:22
and run the model into pipelines using other OTB applications for instance. Okay, that's just a slide on what's TensorFlow computational graph. It's just some symbolic programming. You define some operators and you link them together
05:44
and they process the tensor in a given order from start to end. You can do a lot of things. I won't go deeper in this matter but you can do some very simple model.
06:00
For instance, this is a scalar product. Okay, it's like five lines of code just for one scalar product but do you understand why? You can define two inputs named x1 and x2. Compute the scalar product. So here you just start defining the computational graph. Nothing is executed and you just save a model.
06:21
This is some Google protobuf format and that's all. You just have built the model which is deterministic and it takes x1, x2, computes the scalar products and nothing else. Of course it has a y output which is a result.
06:40
Then using OTBTF you can use this model with two inputs, namely real world images. For instance, two Sentinel images. And you can run the inference, meaning you can run the model over these images using all the complex pipelining stuff of the OTB which deals with memory footprints and stuff automatically.
07:05
So you just create your model and then put it in production, as simple as that. Now we are coming to deep learning. The main problem with deep learning
07:22
is that you can read a lot of cool papers with guys who are motivated by publishing one paper on a big journal. And I think that often it doesn't really work in real life. When you want to apply the model on images
07:42
it is not really simple because guys didn't mention that the processing time is like two days to process a small image, stuff like that. So I'm not really happy, I was not happy with the way I approached deep learning.
08:03
So the idea is to provide the people who apply deep learning on remote sensing images with a set of tools that guide them to create something that works, that really works on remote sensing images.
08:27
So I will detail all that a bit later. So what the features of the OTBTF are compared to the toolbox baseline, there is additional applications for users,
08:43
for instance, for teaching people want to code so they just want to train a model so we give them a few models built the same way I was describing with this color product. And people are just using an OTB application to train the model.
09:02
So you won't go that far with this approach but at least you can learn what deep learning is and how it can be applied on remote sensing images. And for production we have, of course, one single application dedicated to model inference and this one can be used for teaching
09:21
but also in processing a chain in operational centers. We use that in real world production center. I will show some examples in the second part of this presentation. And finally, we have recently this Python class
09:41
dedicated for model training and it targets many data scientists rather than GIS experts like it's the case with the OTB applications. And the goal is to go large scale with distributed training, for instance, on clusters, on the cloud and big datasets.
10:05
So the OTB applications are values, there are some applications to extract patches into images, there are applications to select where you want to extract patches and there is some application to use the model
10:23
to process the images with the model. After that there is a bunch of experimental, I would say, applications, for instance, so you can use classic traditional classifiers that works on features of deep models.
10:40
For instance, you can download a pre-trained image ResNet or something like that, use some features of the network and use some random forest to work on the features of this network. It is interesting, it works, but it's kind of experimental, you know.
11:00
And finally, we have some helpers to extract polygons statistics but it's not really the easy point here. Okay, so why streaming is important? As you know, remote sensing images, time series,
11:20
they are more and more big and you have to process them chunk by chunk. And the OTB has a great foundation to process things like that. The OTB is based on the IDK
11:40
and the IDK subdivides the image into regions and JDAL writes the output in the layout that you have asked for. And typically in deep learning you perform a lot of convolutions so you want to process square tiles. It's better for the optimizer, the processing time.
12:05
So it's very useful to let OTB manage all this writing layout, processing, memory footprint layout and stuff like that and just focus on build a model, train a model
12:22
and just run the model into, why not some complex pipeline like this? This is an example. You have one optical image and one synthetic aperture radar image. You perform some synthetic aperture radar calibration using your photo box. You perform some optical calibration using your photo box
12:42
and you give the output to the model and after that you apply some kind of post-processing like writing the data flag, extracting some zonal statistics from an additional vector layer or things like that. You know there is actually all this part which is very common when you do some operational inference with deep nets
13:07
but I will explain later that you can do that directly inside the model. So it's up to you if you want to externalize it like this and you can do that with OTB for instance or you can do that directly inside the model
13:24
and the model is run on the full image, not just on a small chunk that you have to repeat over and over. It just writes the final image in GDOR-supported formats and that's as simple as that.
13:44
So typically you will extract some patches using the patches extraction application. So you will have some GOT files and you can build a dataset which is TensorFlow compliant from these patches.
14:01
So from here the dataset object will just read on the fly on the file system the images. This is not very efficient but in some cases when your bottleneck is a computational part it's okay, you can just use that. If you want some performance, if you target cloud computing or cluster computing
14:23
you have to use the TF record format and you can just quickly convert this dataset into TF record object and it is quite easy then to use that in a distributed manner. We have extensively tested this framework
14:40
on the Jansez supercomputer of France Institute for Development with, I don't remember, like 100 GPUs or something like that. So it's really performance grade stuff. And the class provided to develop the model
15:03
it can look a bit complex like this but it is very simple. You just have to override this class when you develop a model and everything is already prepared to ease the work.
15:28
Typically, this is an example, I create my model in Python so I import the model base class. I implement my model, so here there is four convolutions and one softmax it's for, I don't know, long cover mapping
15:43
so I implement the get output method and I can also implement some normalization function to scale the input data, things like that. I can also pre-process the dataset for instance, if I want to rename some inputs or normalize the target data or input data.
16:05
And that's all. I can just use the model, the object like that as Keras model. So, you know, it's really simple. So here it's just a minimal example. I don't have put some callbacks, tensor broad callbacks, stuff like that.
16:22
You know, but it just computes some metrics and it optimizes my network using the ADEM optimizer. And finally, I save the model as my Scala product just before and I can use it in the workflow toolbox pipeline from command line like this or from Python.
16:42
This is PyOTB. I strongly encourage you to see the presentation of Nicolas in this afternoon. And the output post-processing is also a crucial point in deep learning at operational scale
17:00
because a few people are talking about the blocking artifacts that are generated in the output images of deep nets. And we provide some default post-processing functions that crop the output tensor in the spatial dimensions just to avoid the blocking artifact. This way you can apply fully convolutional models
17:21
on arbitrary sized chunk and have a smooth image not including blocking artifacts. That's an important point. But you can implement your own, you can just override this method and implement your own post-processing method.
17:47
Okay, so here are a few examples. I will be very quick. These are examples of real work that we did with OTBTF.
18:03
So we mapped the entire France mainland at 1.5 meter from spot six and seven images. It is a building footprint, so there is different kind of buildings, but it's not really pertinent. It was just a large-scale test. And the originality is that we use a model
18:22
that processes separately the panchromatic and multispectral channel at different resolution. So we use two input sources in the OTBTF model, and we produce the level at the finest resolution. We did that also for the... So this is a few nice screenshots.
18:41
We also did that for the forest. So we mapped the whole forest of France from spot six and seven images. So it's the exact same approach. We just changed the target data. And we also worked on super resolution.
19:02
So you can see in the left, it's the original Sentinel-2 image, and on the right is the super-resolution image. The model was trained from spot six and seven images. And we trained over France, but in fact, it works quite well over the world. It is Lisbon.
19:22
Previously it was Amsterdam. And this is Oklahoma City. And this is some Pao2 in China. And it's easy to run. It's a one-liner, and you can also use it if you want to see how it works. You can just call the model like this.
19:40
So this is from the Python OTB API, and this is from PyOTB. So I recall the... Yes, this is tonight, PyOTB. Go take a look, guys. And we also did a great project recently
20:00
with the French Space Agency. We provide pre-trained models for cloud removal in optical images from synthetic aperture radar. And it works. It is just not a nice show, you know. It really works. You can process the image in reasonable processing time.
20:21
Like on a GPU, it's one minute to process one full Sentinel-2 image. And you can even use CPU because this is a small model. But yes, it depends. We have some mono-temporal approach, some multi-temporal approach.
20:41
And the multi-temporal approach, of course, uses time series instead of one single scene, and we have, of course, better results. So this is a four-year, or reconstructed, spot Sentinel-2 images. So it's an area in the south of France. We provide pre-trained model only in south of France.
21:02
It was the location of our study. It's like 10 Sentinel-2 ties in a geographical area. So yes, this approach, I think, works really, really well. It's really interesting. But yes, if you want to read more,
21:22
I invite you to read our paper and take a look on this book. Yes, this is really exciting times because we have a lot of data, and we can do a lot of things with deep learning. And we have also site projects, for instance, rock mappings. This is some problem in France
21:43
where farmers, they break the rocks in their fields, but sometimes they do not have the right because it is a protected natural resource because there is some animals that live here and stuff like that. So some guys from the Latrescope company
22:04
did the mapping and have compared the random forest, the SVM, and the convolutional neural networks using OT-BTF. And they found that the convolutional neural networks were more pertinent. OT-BTF is also used in the Teilhard data center
22:20
to produce the soil moisture maps of the world from Sentinel-1 products. So it is used only as a base component for the TensorFlow in two images apart. And, yes, that's it, guys.
22:41
Okay. I will do a small recap. So what you can do, I didn't put a slide on what you cannot do. So what you can do is use the OT-B applications to create some datasets with patches extraction,
23:00
with patches selection. You can use the original framework of OT-B for sample selection, for instance. You can build models in Python. You can train models either from command line interface using OT-B applications, but I recall that it's really for educational purpose for beginners. If you really want to work seriously,
23:21
you have to develop in Python and use the OT-BTF dataset classes. And finally, you can do some distributed training with TensorFlow 2, which is really great because instead of taking days to train your model, it just takes hours, so it's important. And, yes, finally,
23:42
you can run models in OT-B pipelines, which are streamable from end to end with reduced memory footprint. And, yes, that's it. We have some future work, of course, like improve the Python API. Since this is new, we will improve it a bit over the time
24:02
and maybe add more examples into the documentation and make lighter Docker images, improve a bit the CI because we rebuild each time and it's not really efficient, but we will work on that. It does not concern the user. And we have some limitation on the compilation of OT-B and TensorFlow.
24:23
We have to fix that. And finally, we will work on the integration in PyOTB to really ease the mutual interactions between them by X-rays, TensorFlow, and the OT-B applications. I have finished. Thank you.
24:41
Thank you very much.