We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Cloud-Native Geospatial with JavaScript

00:00

Formal Metadata

Title
Cloud-Native Geospatial with JavaScript
Title of Series
Number of Parts
351
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language
Production Year2022

Content Metadata

Subject Area
Genre
Abstract
The amount of Earth Observation data we have available nowadays is exceeding the capabilities for data processing. Therefore, a lot of data is now made available in the cloud. To make digesting the data easier and more-lightweight, it is getting more and more popular to store the data in so-called “cloud-native” file formats while data processing is also moving towards the data, i.e., into the cloud. This way you only need to retrieve the actual subset of the data you are actually interested in instead of the full data set, which can be in the magnitude of gigabytes or even larger. This technology of cloud-native file formats is usually best used with Browsers, which is the users’ main interface to the internet and the cloud. There the main language is JavaScript. Therefore, this talk will give a high-level introduction about the relevant cloud-native file formats and show whether and how you can make use of these files in client-side JavaScript: - COG: Cloud-Optimized GeoTiff (cogeo.org) - COPC: Cloud-Optimized Point Clouds (copc.io) - Flatgeobuff (flatgeobuf.org) - GeoParquet (github.com/opengeospatial/geoparquet) - STAC: SpatioTemporal Asset Catalog (stacspec.org) - Zarr (zarr.readthedocs.io) This talk will dig into the available open-source libraries and, if JavaScript implementations are available, show their functionality based on examples. If multiple options are available, a high-level comparison will show the main differences in functionality. For COGs for example, we’ll compare the capabilities of the popular mapping libraries Leaflet, OpenLayers and MapLibre GL.
Keywords
Slide rulePoint cloudSoftware engineeringWeb browserGoodness of fitUML
Slide rulePoint cloudWeb browserData storage deviceServer (computing)Computer fileOperating systemLibrary (computing)Stack (abstract data type)GeometryGoodness of fitWeb browser1 (number)Link (knot theory)Field (computer science)Visualization (computer graphics)Connectivity (graph theory)RoutingServer (computing)Slide ruleBefehlsprozessorPoint cloudTesselationWeb 2.0QuicksortSemiconductor memoryObject (grammar)MereologyProjective planeDifferent (Kate Ryan album)Process (computing)CurvatureMultiplication signNeuroinformatikMathematical analysisCASE <Informatik>Point (geometry)Data storage deviceDisk read-and-write headService (economics)Social classSoftware maintenanceLatent heatVector spaceMedical imagingComputer configurationView (database)Functional (mathematics)Library catalogType theoryLevel (video gaming)Partial derivativeReading (process)Category of beingSet (mathematics)Utility softwareMetadataInformationoutputOnline helpImplementationRevision controlElectronic mailing listRight angleMathematical optimizationPlug-in (computing)Computer animation
CodeFile formatPoint cloudBitWeb 2.0Design by contractCodeLibrary (computing)File viewerSoftware testingComputer animation
Array data structureFile formatReading (process)Library (computing)Mathematical analysisSlide ruleBootingSoftwareAxonometric projectionPlug-in (computing)Point cloudShape (magazine)Device driverTelecommunicationClient (computing)Component-based software engineeringVisualization (computer graphics)Interactive televisionText editorStapeldateiFrequencyMonster groupGroup actionWeb browserStack (abstract data type)WordLibrary (computing)Web 2.0Different (Kate Ryan album)File formatReading (process)Point cloudCASE <Informatik>Visualization (computer graphics)Single-precision floating-point formatCanadian Light SourceMappingComputer fileQuicksortExterior algebraException handlingLatent heatClient (computing)MereologyProjective planePlanningConnectivity (graph theory)ImplementationSlide ruleWeb browserComputing platformCurvatureVector spaceOpen sourceTowerInstance (computer science)Stack (abstract data type)TesselationArray data structureData storage deviceMIDIDimensional analysisFitness functionGoodness of fitServer (computing)Object (grammar)Level (video gaming)Shape (magazine)Revision controlGeometryProcess (computing)Operating systemComputer animation
Computer animation
Transcript: English(auto-generated)
Thank you. Good morning. My name is Mathias. That's Daniel. We're both two more or less freelance software engineers working a lot with cloud native and JavaScript and basically in the browser. So It would probably be good if Matt was talking now and we were after because now we need to go
Into the like high level information about what cloud native is. Otherwise that would be his part. So cloud native geospatial basically means that you have very large data sets And you want to store them usually with like setup server or web map service or something like that But in this case you would store them in a
like different way Specific way that makes it cloud native and you can then place it easily on object storage for example And then only download the parts that you actually need that is in the end faster and cheaper both in costs and in time And as such you usually don't need a server, which is called so-called serverless design
So you probably for example, if you have a geotiff on online that is cloud optimized You wouldn't need a tile server like tie-tied or so And what we're going to do today is to try to convince you that you may go this new route
Making stuff cloud native and that you can work actually with all the data available Usually in the browser as well You don't need a server that pre-processes the data for you and makes it available for the browser Like it was in the past We are focusing on JavaScript and typescript and if there's nothing available in that natively then we also look into a web
assembly We ignore node.js for now because that's again server side So quick overview of what we were we are tackling here So that's first the part of discovery which we heard before from Pete is stack Then we're having two categories types of data, which is rest or array based data
Which is cogs cloud of my view tips PM tiles cloud of my star and czar and then for vector data we have geo parquet flat geo buff and thought of my point clouds and For cloud native processing you have open yo
A short disclaimer we're working on a couple of projects that we will mention in this Slide and all the ones that have a star there that is maintained by us So just as a disclaimer that we might be biased in some of them So we heard about stack And stack in the browser very much fields native because it's just chasing right Jason is coming from the JavaScript ecosystem, basically
So reading it is like very easy you don't usually need it really a library for it So there is no general-purpose libraries like pi stack and you can use Stack with something else then Python of course Pete was mostly focusing on Python. I think
so That's good. And you have also like the stack items of you Jason so you can all use every tooling that is around for geo Jason processing there's a couple of Tools that help you to actually Work with the data in more advanced ways. For example If you have like old stack files and want to migrate into the latest version
There's tech migrate which just takes your stack item and collections and migrates into the latest version There's stack fields which converts tech metadata to HTML and all that is then used again in more high level Implementations like for example stack browser open your view components and stack layer
Which all are for visualizations tech browser you've seen before in the stack talk which is for basically a UI for for stack Where you can browse through all the catalogs and API search in them visualize them, etc There is a small utility class in stack browser as well that you might want to spin out at some point
Just with half of a head of functions To get it asset specifically or link or whatever And then stick layer is a leaflet plug-in basically to visualize tech items and collections which also includes coke visualization if there is cloud optimized geotiff in the
Item basically in the assets and view components is basically a web component Library for also for stack items and and collections originating in openio So you can load that basically and then you have HTML text available without like really needing to write JavaScript where you can input your Data and show it just in the browser without any like coding involved except for HTML of course
So this slide is on cogs cog stands for cloud optimized geotiff and cloud optimized geotiff is a
TIFF image with geospatial metadata, and it supports partial read requests Which mean you can grab just part of that image You don't have to read like the whole thing So there's a lot of different options when it comes to reading visualizing and analyzing
We're not going to have time to go over every one But I have a question for the audience so raise your hand if you work with cloud optimized geotiff's oh Wow okay, that's no no no keep them up keep them up Okay now for anyone who didn't raise your hand look around and if you have any questions
You find one of those people and you ask them now or at the icebreaker So yeah lots of options there's low-level options like geotiff JS, which is an amazing library Fabian Schindler is the the lead maintainer there
And eoxis supported it, thank you in a lot of the the reading libraries depend on that there's also Libraries that will try to get gdal into the browser those come with a lot of features But you also have to pay for the the large file size
But you know as computers get faster. It's not it's not unreasonable to check those out Visualizing there's a lot of different options there from sort of more memory optimized CPU based visualization to real high-powered GPU enabled visualization And then there's also analysis
options too But I forgot to say is that there are a lot of names in here and listed a lot of them some have links You can go to foster G to lieutenant at the e and then the slides are there you can just click the links Google the names if there are no link Just so that you don't need to write all these like lists down
Then the next cloud optimized format is clocked up to my point clouds There is a library available That is called copy CJS, which is based on typescript and then has some kind of reader in the background. Which is web assembly Unfortunately for that that is still very limited in the documentation, so there's just one example available no API documentation
So you need to be prepared to dig into the copy siege code or contract the company that is in actually maintaining it So there's a bit more work to be done, but there's also an online viewer that you can use to actually test it out
So czar Is a format for chunked or in other words tiled compressed multi-dimensional arrays you might be wondering What's the difference with cogs? Well czar will have more than one dimension The the tiles will be kind of large, so it's a good fit for climate data
There's it's it's sort of a newer format Geotiffs have existed since the 80s In cloud optimized more recent, so there's still With some of these newer cutting-edge formats. There's more work to be done
But there's a really awesome Step forward that carbon plan did with visualizing czar data On the web, and there's also a couple different efforts in JavaScript So we're looking forward to tracking this and seeing where it goes
Then another one that has recently been published as a specification and still is not really finished But it's it's there and it seems to be the step forward for vector is geo parquet There is nothing specifically implemented yet for geo parquet in JavaScript
But there is like there were two implementations, but geo parquet or parquet itself is pretty Difficult so they have been abandoned in favor of web assembly based implementation Which is parquet bosom And that is also used in Lotus GL
So if you're like a deck deck GL user or want to go into any like GL based with PL based Yeah, visualization then Lotus GL might be the place to go as it's also available for a couple of other Formats that you're talking about like flat geo buff
Flat geo buff is an amazing new format by And I apologize, I'm an American I actually live in the south so you can imagine that the accent it Bjorn Hartell apologies on the pronunciation, but Really, it's find that guy here on github and ask him about it. It's been getting a lot of
Support across the community on a bunch of different projects most importantly gdao Which means you can really use it everywhere and Reading is fully supported in
JavaScript There's there's a lot of different work going on and visualization and reading but sort of the main takeaway would be you can use it to Replace a shapefile and it's cloud optimized Then we have cloud optimized single file map tile archives
Which is basically a replacement for your tile server in the background that can be used on object storage There are two of them that one is PM tiles. The other one is kotar, which is cloud up my star Which sounds rather weird, but is the thing Yeah kotar there is a general-purpose reader for kotar and PM tiles as an implementation for both leaflet and map Libra
and Openio is the last part which is about processing that is a specification to unify all the Processing platforms out there with a specification There's a JavaScript client available. There are realization components and a web
component for editing in a non coding way, basically So there you go. And now the probably most important or interesting slide is in the background now the conclusion like we have listed now all the Like formats that we have available
And give some hints about whether Libraries available that the visualization mapping tool is tooling is available and really would recommend it in the browser and How good the documentation and examples are? so you see there is a couple of really JavaScript implementations as libraries available and if there's not there is a
web assembly Alternative available so you can all use all these cloud of my formats in the browser For most of them there is also visualization available except at least in an open source Way, not for cloud optimized pong cloyds kotar and geo parquet right now
and for those are depend very much on the use case and What we would recommend in the browser right now without any doubts is a lot of nice geo tips kotar PM ties and stack And for the others, it depends usually on the use case like for example
Cloud on the spot clouds. I said the documentation is lacking So there's something that you still need to like you would need to dig into the actual Library and understand it deeply. Otherwise, it's hard to implement it and get it running For the jet flat geo buff. We have for example the issue that the files get large You don't have overviews like in Cox, so you
Can't like get broader more generalized instance of your vector data For Gia parquet, it is implemented but the geo part is missing in the library So there is no no special handling. You still need to add the geo part to the to the parquet version library Openio, it depends on the use case and for czar there is an issue that
Depending on how it start the chunk size is an issue like for some as our fights chunks are very large So it doesn't really it's handled well in the browser But if you can manage to optimize that to chunk size that are Small enough for the browser then it actually works pretty well
Yeah, that's it I think right yep