State of GDAL
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43447 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Bucharest 2019137 / 295
15
20
28
32
37
38
39
40
41
42
43
44
46
48
52
54
57
69
72
75
83
85
87
88
101
103
105
106
108
111
114
119
122
123
126
129
130
131
132
137
139
140
141
142
143
144
147
148
149
155
157
159
163
166
170
171
179
189
191
192
193
194
195
196
197
202
207
212
213
214
215
216
231
235
251
252
263
287
00:00
Level (video gaming)Software developerOpen sourceProjective planeMathematicsSlide ruleKey (cryptography)Independence (probability theory)Computer animation
00:37
Open sourceLibrary (computing)AbstractionVector spaceRaster graphicsReading (process)File formatCommunications protocolSeries (mathematics)Device driverASCIIShift operatorGoogle EarthBinary filePRINCE2SoftwareFile formatWeb browserSequenceRevision controlINTEGRALBitCycle (graph theory)MathematicsGoodness of fitSlide ruleService (economics)Device driverLine (geometry)Exception handlingMedical imagingWeb 2.0Device driverServer (computing)Communications protocolGradientRow (database)Open sourceTwitterSource codePartial derivativeMappingFlow separationRaster graphicsData storage deviceVector spaceSoftware developerElectronic visual displaySet (mathematics)Software testingProduct (business)DemosceneProjective planeWebsiteProcess (computing)MereologyComputer animation
04:10
Device driverGoogle EarthDevice driverImage resolutionVariable (mathematics)CodecPolygonKeyboard shortcutSuite (music)Human migrationLine (geometry)Linear regressionSoftware frameworkSoftware developerCore dumpCodeService (economics)Client (computing)Computer-generated imageryVector spaceGroup actionComputer configurationProbability density functionGroup actionDot productData compressionEmailPresentation of a groupProjective planeRight angleCASE <Informatik>Device driverGame controllerComputer configurationAsynchronous Transfer ModeSatelliteElectric generatorSquare numberElectronic mailing listNumbering schemeDampingSoftware maintenanceDimensional analysisImage resolutionCybersexWeb pageDigitizingTerm (mathematics)Error messageDescriptive statisticsVideo gameMachine visionMaxima and minimaLine (geometry)Execution unitExistenceCellular automatonDevice driverCustomer relationship managementVideoconferencingGoodness of fitDatabaseRoutingDifferent (Kate Ryan album)SoftwareFlow separationSoftware testingSoftware developerBit rateRevision controlPhysical systemProcess (computing)Arithmetic progressionSoftware suiteSineService-oriented architectureLibrary (computing)Interpreter (computing)WritingPredictabilityMedical imagingPhotographic mosaicVector spaceQuery languageSet (mathematics)Variable (mathematics)Open source1 (number)Suite (music)InterpolationSingle-precision floating-point formatRaster graphicsCore dumpComputer fileSoftware frameworkCodecData storage deviceAlpha (investment)Key (cryptography)PixelKeyboard shortcutArray data structureJava appletWeb 2.0VirtualizationLocal ringInstance (computer science)BuildingUtility softwareFitness functionPolygonAlgorithmLinear regressionRepresentational state transferData acquisitionFile systemLink (knot theory)DemosceneLecture/ConferenceComputer animation
12:04
DisintegrationComputer reservations systemCoordinate systemOperations researchDefault (computer science)DatabaseWebsiteFunction (mathematics)CodeReading (process)Probability density functionWeb pageExecution unitRule of inferenceTablet computerMenu (computing)Programmable read-only memoryInflection pointMaxima and minimaDevice driverHierarchyVector spaceArray data structureLink (knot theory)Raster graphicsCumulative distribution functionSoftware frameworkUtility softwareWeightProbability density functionArray data structureComputer fileCodierung <Programmierung>Coordinate systemDevice driverMathematicsDatabaseLevel (video gaming)Web pageComputer configurationImage warpingSet (mathematics)Right anglePoint cloudOrder (biology)Software bugReading (process)Medical imagingRevision controlVector spaceConfiguration space1 (number)InformationDevice driverMathematical optimizationWebsiteMereologyOperator (mathematics)Projective planeSpring (hydrology)DampingFilm editingComputer reservations systemAreaData structureAdditionQuery languageFluid staticsNumberStandard deviationMultiplication signSoftware developerWeb 2.0Modal logicLibrary catalogINTEGRALLink (knot theory)Flow separationGeometryDefault (computer science)Content (media)TelecommunicationHierarchyDimensional analysisClosed setMappingState of matterCuboidComputer programmingCodecNetwork topologyBitEstimatorPredictabilityMachine visionCondition numberNeuroinformatikSinguläres IntegralTime zoneDifferent (Kate Ryan album)Type theoryDataflowTraffic reportingCoefficient of determination19 (number)SurgeryQuantum stateComputer animation
19:55
Different (Kate Ryan album)Electronic mailing listEmailComputer clusterMultiplication signDirected graphComputer animation
20:41
Projective planeDifferent (Kate Ryan album)Standard deviationMereologyData conversionSlide ruleCodeChannel capacityDecision theoryNichtlineares GleichungssystemPhysical systemBuildingCASE <Informatik>Singuläres IntegralPoint (geometry)Coordinate systemDatabaseProcess (computing)Integrated development environmentOntologyGoodness of fitWeightRaw image formatLecture/Conference
23:24
BitWorkstation <Musikinstrument>Hand fanMultiplication signExpected valueLecture/Conference
24:18
Keyboard shortcutMultiplication signSoftware developerSingle-precision floating-point formatLecture/Conference
26:04
Lecture/Conference
Transcript: English(auto-generated)
00:07
Thanks, so welcome home. My name is Éven Roux, I'm an independent of free and open source software developer mostly focused on GDAL, Map Server, Proj, LiveGeotiff, QGIS and a few other projects.
00:27
So in this talk I'm going to give an overview of the changes GDAL has received during the last year with 2.4 and 3.0 releases. So just to sum up GDAL in one slide, so GDAL stands for Geospatial Data Abstraction Library
00:45
and it's a piece of software you use sometimes without even realizing it. When you want to read or write a geospatial format in most C or C++ open source or closed
01:02
source GIS software. So as of today it handles more than 240 different formats and as a trend of recent years it also handles more and more network protocols and services. GDAL uses MIT X open source license which is a permissive one enabling you to use it
01:25
in all software you want. So over the course of last year we had two feature oriented GDAL releases which is quite unusual because we generally follow roughly yearly annual cycle but as the integration
01:46
of Proj 6 turned out to add some backward incompatibilities we decided to do an unanticipated GDAL 2.4 releases with good stuff that was already ready.
02:02
And so even if it was a seven months development cycle it was a fairly busy release with nearly 1,400 commits. So each new GDAL version comes with new drivers and 2.4 was not an exception to this.
02:26
I will not go into much detail about the free items on this slide because those are mostly side products of the Proj 6 effort and technical drivers to read Datum shift grades.
02:43
An interesting new item is the NGW driver, NGW stands for NextGIS Web and it's a server GIS which allows you to store and edit geodata and display maps in web browser. It's licensed under GDAL 2 and we have also this EEDAA which stands for Earth Engine
03:08
Data API Image Driver and it's a way of accessing rasters which are stored on Google Earth Engine with partial download capabilities of piece of images.
03:23
So the Earth Engine API itself was quite a bit experimental when the driver was released so make sure you use the latest GDAL 2.4.2 release which has received the upgrades for the API changes. On the vector side we have the GeoGISN sec driver, so sec is for sequence of GeoGISN
03:48
features which are separated by a new line or record separator characters and so this is a cleaner way of having big data sets of GeoGISN instead of having a gigabyte GeoGISN
04:05
feature collection which used to blow your RAM when you tried to open it so GeoGISN sec is an appropriate tool for such huge data sets. I mentioned NGW on the vector side and it's raster side counterpart too and the same
04:25
for EEDA, it's a vector driver which enable you to query which rasters are available on Google Earth Engine and do a special or attribute queries to find the image you
04:42
are interested in. Some drivers, existing drivers have received dates of the bag, bag is for bathymetric attributed grid and so now the new version of the driver can support reading variable resolution grids because bag comes generally from a lower resolution grid and several
05:06
chips of different resolution and the driver can do an interpolation of that to present a single resolution grid and write support was also added. Regarding the GERTIF driver, it received support for two new codecs, LERK stands for
05:29
Limited Error Raster Compression, it's a lossy compression scheme where you can define the maximum amount of errors in terms of pixel values you want to write.
05:43
And the WebP codec is also useful in use cases where you would have used a JPEG before and one of the key feature of WebP is that it supports alpha channel and so this is quite a promising codec if you want to store big GERTIF data in a lossy way.
06:08
The PostGIS raster driver received support for a database raster, so those are raster where the imagery is not stored in the database itself but in a file on the file system and
06:25
the driver has been optimized so that if you run it against a local PostgresQL instance it can actually short circuit database access to directly read the file from the file system.
06:44
We also added two new virtual file systems for the ADUP file system, so VSI HDFS uses lib HDFS library which is a C library on top of a Java library and VSI Web HDFS uses
07:07
the REST API for HDFS, so the last one is a built-in for standard builds of JIDAL. The JIDAL control utility was mostly rewritten from scratch, it uses a marching square algorithm
07:24
and it's much faster than the previous one and it also received a new mode to generate polygons whereas previously you could only generate control lines. And we also added a few, I would say, anti-features, the removal of the PHP and Ruby bindings
07:44
which didn't receive any maintenance so they are gone, if any of you want to bring them back to life so please join and take the stick.
08:01
This item is mostly of interest for JIDAL developers themselves, so JIDAL has a huge regression test suite which is made of more than 200,000 lines of Python, so by itself it's bigger than some software projects and it was written in a kind of Hadoop test
08:26
framework so it was really unfriendly for newcomers and even for core contributors it was quite hard to write tests. So Craig, the sticktor, volunteered to port the existing
08:41
test suite to use a pytest framework and it was a huge work, he used a bowler Python library which enabled him to do most of the way in a semi-automatic way and he gave a presentation about how he did that at Python Australia and included the other link
09:03
to his talk and so this was a really massive effort and really curious to him and his employer for leading this effort. So let's come to JIDAL 3.0, so if you followed the mailing
09:21
list at the beginning we considered calling it JIDAL 2.5 but as there were some backwards incompatibility and purge 6 was actually a new requirement we decided to call it for JIDAL 3.0 to really indicate that there was a new step. A few new drivers
09:46
have been added so there's this DAS driver which is to connect to an Airbus satellite imagery so it's quite similar in its purpose to other similar drivers like the RDA driver
10:01
of DigitalGlob or the Planet Pale Scene or Pale Mosaic driver or the Earth Engine data API image driver I just mentioned before. We also have a new driver for TileDB so it requires JIDAL to be built against libTileDB open source library and TileDB is a way
10:26
of managing massive dense and sparse multidimensional arrays. On the vector side there's a new driver which is the latest version of the Limb Mongo CXX connector and this enables
10:45
you to connect to the latest version of the MongoDB database. We also add a few improvements in existing driver so the NetCDF driver was extended to be able to fetch data sets
11:03
which are organized in hierarchical groups. The PDF driver on the right side was improved to be able to generate a PDF file from an XML description file where you configure your
11:22
layers and which ones are exclusive of each other and it's going to be used by QGIS 3.10 whereas the layout manager has been updated to use this new capabilities. And for those who
11:43
are interested in planetary data sets the last two items the FITs and PDS4 drivers have been also updated with new capabilities. So as JIDAL gets bigger and bigger of their years some people have requirements to make it
12:03
fit in the smaller builds. So we have for Unix builds we have a way to disable all optional drivers and you can manually request the driver you want to build to add in the build or you can also disable some particular drivers.
12:24
There were many Docker images available in the wild for JIDAL so we have added a few official ones if I might say so there are five different configurations ranging from very small builds
12:40
which fit in like 20 megabytes to a full build based on Ubuntu. So now let's come to the main driver for the JIDAL 3 version which is the integration of approach 6. I'll give this afternoon a dedicated talk on all the work that has happened in that
13:05
area so I'm just going to sum it up shortly. So up to now WKT for CRS used to be in JIDAL and that support has been moved down to approach 6 and in the meantime I wrote a move but
13:24
it was essentially rewritten from scratch because we added support for WKT to 2015 and the latest 2019 revision and the fact that JIDAL support them now will enable the wider deployment
13:43
of WKT to which was quite blocked by JIDAL not supporting it. The CRS database which used to be collection of CSV files is now SQL 8.3 database which is under the approach and so this enables to
14:06
to have more query capabilities and be able to handle several CRS catalogs. We also have support from time dependent coordinate operations and the OGR to OGR and JIDAL warp utilities have
14:25
received a new switch the CT for coordinate transformation where you can specify your particular approach pipeline so this is something you you will use typically with a new approach info utility so this is clearly for advanced users that want to precisely control the coordinate
14:44
transformations they want to do on the data and the last item is yeah I guess some people will say it's more a bug than a feature but on the low level layers of JIDAL the coordinate transformation API is now EPSG axis order compliant by default but for the upper layers we actually
15:08
added the necessary magic to not try to break too much existing process. Another non-programming
15:21
activities are an overall of the website and documentation. Up to now the documentation used to be a mixed of doxygen files and static HTML pages it does quite a none of your structures and it was hard to attract contributors to it so during the Minneapolis cut sprint this spring
15:48
I, Howard Butler, Matthew Sluss, Scott Dan Best and a few other people decided to to challenge this and and convert the existing material to restructure text and use a Sphinx framework which had been used for other
16:04
front projects like Proj, Pudor, or Mater so a large part of this could be automatically converted but we had quite a lot of content that had to be manually migrated so it was a pretty busy week.
16:23
So yes how it looks now so on the left side you have a trace structure where you can navigate the documentation and in the main page you have direct access to the driver documentation. There's also a link to the corresponding PDF and we were quite surprised to to discover that
16:45
it was more than 900 pages long and we have omitted the API documentation from it. So there's quite a lot of documentation and also there's this convenient edit and
17:02
data blink you can see on the top right of each pages so if you notice some typo you know easy things to to fix you can directly click on that and do the correction in a pure web workflow and if you have a pull request then yeah you can upgrade easily the documentation.
17:29
So to to conclude I give you a some preview of GDAL 3.1 so in the development version you
17:40
can already find a new driver for generating cloud optimized geotiff so it's mostly some syntactic sugar to be able to to create a cog with the right set of geotiff options. On the low level we made a few changes for to generate a more efficient
18:08
layout of the geotiff files and we made optimization on the reading side to reduce the number of HTTP GET requests so I think it's now close to the optimum you can get with using
18:24
a compatible geotiff file. Another major work is the addition of a complete new API to be able to deal with hierarchical and multi-dimensional arrays. GDAL has been from the beginning really a 2D raster framework and when we want to deal with
18:47
higher dimensional arrays we had to use this sub data sets concept which is kind of messy and not necessary standard among drivers so now we have a proper API to be able to deal with
19:04
this multi-dimensional arrays so it's implemented for net cdf, hdf4, hdf5, grib. I think the tiledb driver will also be upgraded to use it and I think we we can have more more drivers using this new API. And the net cdf driver on the vector side
19:26
has also been updated to be able to implement the new cf 1.8 simple geometries convention which is a way to to encode vector features in a net cdf file and that's it. All right thank you
19:57
for all your all your work on this Evan. We do have time for some questions. Evan is actually
20:03
Evan's quite responsive on the GDAL mailing list but now's your chance to ask him something in person. Sorry can you repeat the last part? I don't think it will I really have difference
20:44
on the performance side it's more on the on the capacities well actually on the coordinate conversion part which is the main difference between GDAL2 and GDAL3
21:04
setting up a new coordinate transformation can take a bit longer than before uh that's actually the subject for my afternoon talk because now we use what is called a late binding approach so when you want to transform between a coordinate system to another one
21:26
we have to actually do lookup in the database to find the best transformation path whereas before the transformation always went to wgs84 so it was out coded and it was very fast
21:43
but of course there were a lot of cases where this didn't work.
22:08
Yeah that's that's a question that yeah can say over and over um right yeah there are there's actually a project which is called BORT which is a CMake approach to build GDAL
22:26
uh yeah it's it's a side project it's not it's not an official one uh well we have already two build systems and adding a third one to the equation you know it's like this slide
22:44
we have 14 standards and now we have 15 standards so yeah it's it's an open question maybe at some point we we'll have a CMake official build system but adding a third one
23:02
and given the size of the GDAL code base it's quite a difficult decision to take.
23:32
So your question was about how the fundraising went is that it?
23:41
Do you think that the fundraising was easier or more difficult than you had expected? Uh I didn't really have the expectation because it was the first time we tried this so in the end it went pretty well I mean I think in one month we we gathered all the funds so
24:02
I think it was not that bad yeah actually we I must really thank Howard Butler because he really pushed to the funders to to make them give them money that's an open question
24:27
we we must find first big topics like that which can attract what topics where a single funder cannot generally afford
24:41
funding the effort where you need a bunch of people gathering together.
25:15
I'm clearly not a C-Short developer myself so we have a Thomas Seycrest which is within
25:23
charge of the bindings so yeah from time to time he updates it but yeah if you if you have particular needs it's probably better to indirectly connect to him and and see if if you can update the binding or bring yourself your contribution to the binding of course.