The benefits of COG (Cloud Optimized GeoTIFF) outside the cloud
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 351 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/69107 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Firenze 2022137 / 351
1
7
13
22
25
31
33
36
39
41
43
44
46
52
53
55
58
59
60
76
80
93
98
104
108
127
128
133
135
141
142
143
150
151
168
173
176
178
190
196
200
201
202
204
211
219
225
226
236
242
251
258
263
270
284
285
292
00:00
Server (computing)Grass (card game)Spatial data infrastructureComputing platformMathematical analysisSoftwareGeometrySource codeServer (computing)Open sourceRaster graphicsMathematical optimizationRandomizationTesselationComputer fileCASE <Informatik>Computing platformMathematical analysisCovering spaceUsabilityStructural loadLine (geometry)Context awarenessBand matrixMiniDiscPhotographic mosaicProduct (business)CollaborationismWordComputer animation
01:20
Musical ensembleControl flowDevice driverComputer configurationScale (map)Web serviceFile formatPairwise comparisonGame controllerDevice driverWeb serviceOrder (biology)Multiplication signOpen setSoftwareRadiometryBenchmarkComputer fileServer (computing)Row (database)Form (programming)Computer configurationZoom lensProduct (business)Computer animation
02:39
Regular graphSource codeClient (computing)MereologyRange (statistics)Server (computing)Self-organizationContext awarenessComputer configurationData compressionGraphical user interfaceComputer networkParsingLie groupDirection (geometry)Raster graphicsExtension (kinesiology)InternetworkingGreen's functionMusical ensembleContrast (vision)Gamma functionForceData compressionSet (mathematics)VideoconferencingStructural loadComputer configurationWebsiteServer (computing)Projective planeContext awarenessComputer fileDatei-ServerRemote procedure callBenchmarkMereologyConnected spaceWeightMixed realityComputer animation
04:07
Contrast (vision)Green's functionSuccessive over-relaxationNormal (geometry)Asynchronous Transfer ModeGamma functionApplication service providerPortable Object AdapterForceEvolutionarily stable strategyRaster graphicsExtension (kinesiology)Musical ensembleZoom lensInterior (topology)AreaPixelSquare numberInternetworkingConnected spaceOffice suiteComputer animationSource code
04:32
Server (computing)Instance (computer science)Electronic visual displayCache (computing)Computer-generated imageryRaster graphicsSource codeData storage deviceServer (computing)GeometryProxy serverObject (grammar)Computer fileRenormalization groupCache (computing)Set (mathematics)Medical imagingSoftware testingLevel (video gaming)Multiplication signComputer hardwareBoundary value problemMereologyCASE <Informatik>Source codeTesselationProcess (computing)Data storage deviceRegular graphGoodness of fitPlug-in (computing)Raster graphicsWeb 2.0Streamlines, streaklines, and pathlinesMiniDiscBefehlsprozessorComputer animation
06:51
Data compressionBenchmarkBenchmarkData compressionServer (computing)Computer fileData qualityGeometryBefehlsprozessorFormal grammarWeb serviceRow (database)Different (Kate Ryan album)Computer configurationReading (process)RepetitionWeb 2.0Software developerInsertion lossMultiplication signDefault (computer science)Computer animation
08:38
Cheat <Computerspiel>Device driverLine (geometry)Universe (mathematics)Source codeData compressionMetreComputer animation
09:04
Radon transformAlpha (investment)Digital signalDiscrete element methodData modelEndliche ModelltheorieData compressionDigitizingComputer fileSurfaceVirtualizationComputer animation
09:36
Random matrixCone penetration testMalwareVarianceComputer hardwarePixelData compressionReading (process)Server (computing)Computer fileSatelliteBound stateOpen setSemiconductor memoryWeb 2.0NeuroinformatikDevice driverData conversionValue-added networkFreewareForm (programming)Translation (relic)Software development kitSlide ruleSource code
11:39
Computer configurationData compressionMusical ensembleSatelliteAxiom of choiceBilinear mapComputer-generated imageryAverageContext awarenessSource codeFile formatServer (computing)Mathematical analysisProcess (computing)Electronic visual displayPersonal digital assistantBefehlsprozessorMiniDiscCache (computing)SpacetimeComputer fileRaster graphicsMathematicsProcess (computing)Server (computing)Multiplication signLevel (video gaming)Product (business)Moment (mathematics)SpacetimeBilinear mapLine (geometry)FreewareSoftware testingAxiom of choiceProxy serverContext awarenessImage warpingBuildingComputer fileData compressionPairwise comparisonCartesian coordinate systemElectronic visual displayRaster graphicsException handlingSource codeSystem callConnectivity (graph theory)BenchmarkMedical imagingEqualiser (mathematics)Computer configurationCodeDifferent (Kate Ryan album)Complex (psychology)Range (statistics)Video gameComputer hardwareDegree (graph theory)SatelliteRepository (publishing)BefehlsprozessorType theoryCache (computing)Row (database)Mathematical analysisAlgorithmView (database)Compass (drafting)NeuroinformatikTouchscreenOpen sourceComputer animation
Transcript: English(auto-generated)
00:01
Okay, so as Nicole I said I will present the benefits of the cloud optimizer teeth outside the the cloud So if you want about me as Already said I work from a geo geo to France geo de France a collaborative data platform in all the France And the special data infrastructure is powered by geo orchestral. It's a very some talk about it
00:26
So if you if you want to know more do not hesitate and in I'm in charge of for long cover on unused data production and analysis so is the region between Leland Paris for
00:41
The use case what the pandemic context because we need to maintain the efficiency of the G's departments and so load quickly a large amounts of Rasta data by VPN with bandwidth issue on the random speeds and Also, the other use case was to trim line Optimized the disk usage because for one data source we have
01:05
some raster tiles pre calculated for Geo server publishing on the SDI and also some raster mosaic files with overviews For GIS software because we have also mixed proprietary on the open source
01:20
Software So we made the first benchmark how how to deal with that? The bill the so we've OGC web service and also the five former we usually use so The advantage of the w ms and w NTS was displaying fast But you do not have a bond order control or stretching to play with radiometry or colorimetry on your roster
01:45
So it wasn't enough Efficient for all the services, so we we will see about WCS which is good because it's like if you're loading the raw data So you have bond control on stretching option, but it was too slow for for us. We lost in productivity
02:07
and about loading directly the raw files, so usually deliver in gp2 or jitif It the duty was fine, but we have to wait to download all the data before the first loading
02:22
But after zoom and pan was quite fast But it was it taking too much time and gp2 is clearly unusable if you want to use it with the open Gd car open open driver. I mean if you don't have a kakadu driver So what about the cog I've heard it in the first one G at
02:44
2019 So I decide to to interest and so it's the definition from the official project website To do as It's the streaming principles so you you can load
03:01
heavy data large data set without Wait to download all the data sets so Like a video a video streaming or you want to check apart, and you don't have to wait to download all the older data so Cog with the cloud in the lockdown context we want to do some benchmark and the only way to generate some
03:25
Cog was to process on the remote computing server without any graphical interface, so but after we need to check if there is a Corrupti no corruption. No and all the the compression option available was was simply so I
03:42
Just want to drag and drop from QGIS from my remote network drive And I didn't expect it was as efficient I imagine I can have a tea on the breakfasts because the the files made 250 Gigabyte which is quite just with only connection and as you see when I drag and drop it immediately
04:06
loading so on 32,000 square kilometer which is quite large area with a pixel size of 20 centimeters, so as you see I can zoom and pan without any problem just with my
04:20
home internet connection from the remote drive from the office so As it answer to your needs about the desktop parts
04:42
We try to continue to see how the cog is is reading by the over Server side so in QGIS without any cache settings or any map proxy for example The WMS and the billion tears was displaying really fast So it was good news and from geo server. We loaded I precise for geo server user
05:05
We didn't have yet test on the cock plug-in available for geo server, so we loaded as jutef jutef raster data source in the server server store so as a regular tify and
05:21
Also, the performance was really amazing because it displaying as fast as the image pyramid cached in the in the SDI So it was also good news for our IT department because we Don't need to ask a new HTTP server or S3 bucket object storage to add to our
05:43
Infrastructure it so also work with our G's desktop, but it's not as fast as QGIS one So if you remember the the two use case we need also to a streamline disc usage and On the SDI we for one data source we we need nearly add free
06:03
duplicate data because you have pyromet tiles for the geo server also with the web cache and Some road raster files to load it so for Python processing for QGIS desktop on some WPS such of extra tool because on the SDI if you want for example to
06:23
Download an auto photography on your territory you will select and it will clip on your boundaries, so we need to keep free nearly free three times the data and after Everything is from cog With good performance and even cache is not necessary it depends of your hardware
06:46
Availability because it can consume some a little more CPU But There is not one one cog in fact you have many compression option if you read the digital documentation So we try to do benchmark as the default delivery former for example was the GP to but offer
07:06
Depends how he had been made both lossy or lossless compression But as you see the the size is is very low compared to the raw data so we test different compression method and
07:21
The repeat the web P is clearly a good clue because you have a very efficient compression we test to develop quality for JPEG and with two because if you compress GP to files Uncompressing in time and then recompressed in JPEG and web P you will compress twice
07:46
So you will loss data quality so from raw data We we can use compression Higher with 75, but if the raw data come from GP to we use
08:01
90 for quality to keep the same quality but Even if the the web P is seems very efficient for compression size It's not very Performance for during the reading so from QGIS or even from
08:21
Geo server as as web service, so the JPEG and deflate one was clearly the most For the most performance reading I'm displaying in the different G's web you've all all this stuff so
08:40
So I try to share some Experience we've made to convert because it depends from your source. You you cannot use the There is not you know universal Compression method you will have to deal with two or three meter depending of your data source
09:04
So for one bond roster such as a digital surface elevation model surface or elevation model We will do in two step first make a via T from the ask ask files and
09:21
then convert directly to to cog and We use deflate because it was the the the better Displaying compression method and If you if you want to take to make a cog from for auto imagery or virtual imagery
09:43
Free bonds roster only then as you see in the previous slides JPEG is the most is the the bets the best compression method so But we will do in the first step. We will uncompress before on converts each GP to files
10:03
to TIF and then build a VRT and From the TIF obtained and then translate to code why do not build a VRT from GP to and convert directly to Koga as I told you I we use a computing server and
10:24
so the license of the ARDAS SDK is read-only on desktop and not on server and Also, we've got some memory issue if you want to do it for example form Your desktop computer was quite strong with
10:43
hardware we have we've got some memory issue, so The safer method was to uncompress before all the GPS with the open GDK open GP to driver And we use a compression of ZDSTD what it offer
11:01
A best best compression ratio van van deflate for this temporary step also, we see that try to convert translates VRT from GP to files directly to cog we
11:21
We observe some artifact or pixel coruts. So again the faster It's not the faster way, but is the safer So for large data You you will spend perhaps one two or three days, but you don't have to to do it again
11:40
for satellite imagery's but Get more than four bonds. You cannot use JPEG because it's only a RGB For free bands, so you can you can use as a destiny, but the destiny is not well well compatible for all G's application we use so we use deflate as the safer option
12:10
Some of a good practice you could you could be Trying to use a GDAL warp instead the two-step GDAL build VRT and then GDAL translate
12:20
But again GDAL build VRT and then GDAL translate is more efficient than directly use GDAL warp if you got a Several files As I told you previously as GP to all I said W is already compressed to avoid Image degradation do not set a compression level to too low
12:44
But if you start from native type native row type, then you can set a different comparison level Try to to gain and if ZSTD is not working you can use deflate without any any issue I
13:01
In the common life. I use resampling method Bilinear it's clearly depending of your user choice on your practice because in my previous job they prefer average, so It's it's it's not it's not a problem to chance rest. That's that's that option. I'm going too fast
13:31
The call gopher an open-source Suitable from any geospatial Application even outside of a cloud context. So without an HTTP server on
13:43
Astra as free bucket and It's it's a very amazing performance even of Samba share Cifs or NFS my my test was from SFTP server When I do the screencast, so it sucks also it improved display time, but also the processing time
14:04
For example, we were computing each month some viewshed analysis on the wind turbine on all the region and When I use the the row DM in in type even with overview It it takes more than eight hours and I directly use the the cog and execute the viewshed algorithm
14:26
It takes less past six hours. So it's it's also efficient for processing. So it's not it's not Timeless to first convert and then processing it As you see there is many option many compression options
14:44
So it will allow you to convert and find the right cog to a wide range of raster data sources But don't don't forget the coke files with Equal quality will already always be a bigger than gp2 on or say the value so
15:06
you have to you have to manage this space for that and If you from the GIS server side If you use directly the code you will use a little more CPU as the it
15:25
As if you do not use more anymore than a cache so it depends of your hardware infrastructure you will may adapt that perhaps doing again a cache for some large coke and
15:42
About the compression method it depending of degree of complexity we you waiting but That's why in benchmark and what compression method we retain we we do not use that is today except for the first step to convert to uncompressed gp2 to tough to tiff to tiff files and
16:05
JPEG Excel is seems also very interesting. But again, we will have some compatibility issues with all our GIS components so We didn't retain at the moment for production
16:21
Thank you and all the common line I've put is available on this github repository don't so do not hesitate to ask some question if you if you've got Thank you very much