The STAC JavaScript Ecosystem + CNG Excursion
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 266 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/66569 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSS4G Prizren Kosovo 202335 / 266
10
17
23
44
45
46
47
48
49
50
52
53
80
84
85
91
110
116
129
148
164
167
169
173
174
181
182
183
186
187
199
202
204
206
209
215
241
248
265
00:00
Slide ruleElectronic meeting systemPoint cloudFile formatStack (abstract data type)XMLUMLComputer animation
00:31
CodeSocial classObject (grammar)Keyboard shortcutClient (computing)Field (computer science)Common Language InfrastructureServer (computing)Stack (abstract data type)Open setArtistic renderingStack (abstract data type)CodeBuildingWeb browserKeyboard shortcutClient (computing)Connectivity (graph theory)MetadataFunctional (mathematics)MappingWeb 2.0Open setInterface (computing)Validity (statistics)Computer clusterArithmetic progressionFluid staticsGraph coloringLatent heatProcess (computing)MereologyRevision controlInformationView (database)Library (computing)ImplementationField (computer science)CodeWeb applicationDefault (computer science)Point (geometry)Server (computing)Right angleWordCartesian coordinate systemLibrary catalogObject (grammar)Category of beingLink (knot theory)Filter <Stochastik>2 (number)Level (video gaming)CuboidComputer animation
04:05
Human migrationThumbnailMathematicsElectronic mailing listBasis <Mathematik>Single-precision floating-point formatMusical ensembleRevision controlComputer wormLimit (category theory)Field (computer science)AreaImage resolutionGreatest elementCASE <Informatik>MassMathematical analysisForceMeta elementCloud computingPhysical systemSource codeBuildingSocial classManufacturing execution systemMusical ensembleFile formatLibrary (computing)Category of beingMassRight angleLink (knot theory)File systemValidity (statistics)Boolean algebraElectronic mailing listSheaf (mathematics)Group actionExtension (kinesiology)Uniform resource locatorVisualization (computer graphics)Stack (abstract data type)Field (computer science)View (database)Web browserCodeConnectivity (graph theory)Computer fileBitDefault (computer science)RankingFlow separationRootMereologyLevel (video gaming)Interface (computing)Server (computing)Open setVolumenvisualisierungFluid staticsFunctional (mathematics)Instance (computer science)SpacetimeGeometryCuboidCloningSocial classLatent heatDirection (geometry)Object (grammar)Core dumpMetadataThumbnailNumberScaling (geometry)Normal (geometry)Online helpDrop (liquid)HistogramRevision controlClient (computing)InformationYouTubeMultiplicationUser interfaceFormal languagePoint (geometry)Artistic renderingLibrary catalogStatisticsAdditionElectric generatorContext awarenessStability theoryMultilaterationEquivalence relationMaxima and minimaParsingComputer animation
13:24
Point cloudComputer fileLibrary catalogRaster graphicsArray data structureVector spacePoint (geometry)Software bugReading (process)Mathematical analysisDefault (computer science)Interface (computing)Musical ensembleStatisticsDesign by contractCodeLibrary (computing)BootingAxonometric projectionWeb browserVisualization (computer graphics)CASE <Informatik>Web browserLibrary (computing)Web 2.0TesselationMappingMultiplication signLevel (video gaming)GeometryImplementationLatent heatBitExterior algebraCurvatureSubject indexingWritingData compressionOpen sourceCodePlug-in (computing)Interface (computing)Visualization (computer graphics)Goodness of fitStatisticsMathematical analysisUsabilityAreaMusical ensembleSign (mathematics)CodePoint cloudOpen setReading (process)File formatComputer fileMereologyDifferent (Kate Ryan album)SoftwareTable (information)Constraint (mathematics)TowerSemiconductor memoryRevision controlYouTubeAuditory maskingScripting languageSoftware maintenanceSlide ruleDefault (computer science)CuboidArtistic renderingResultantMaxima and minimaPrototypeRight angleBinary fileQuicksortVector spaceMathematicsComputer animation
20:05
Slide ruleComputer fileTable (information)Endliche ModelltheorieRight angleExtension (kinesiology)WordCurvatureEmailWeb browserBitAddress spaceType theoryNormal (geometry)Musical ensembleLatent heatState of matterFile systemDifferent (Kate Ryan album)Projective planeCodeMotion captureAreaRepository (publishing)Context awarenessPoint (geometry)Set (mathematics)NeuroinformatikNatural numberExterior algebraSatelliteScaling (geometry)BuildingVector spaceLibrary catalogGeometryStack (abstract data type)Multiplication signRobotMechanism designElectronic mailing listPhysical systemHypermediaComputer animation
Transcript: English(auto-generated)
00:08
Hello again, it's me again, so yeah, so I'll be talking about the Stack JavaScript ecosystem. As I didn't go as far with implementing all the JavaScript stuff, I also added a small excursion into like how
00:22
to do JavaScript with a cloud native file format. So that will be at the end of the talk as well. So just a quick overview of what is out there in JavaScript for Stack. So we have, of course, browser-based in JavaScript,
00:41
and we have Node.js for server-side, and then there are still also TypeScript in there, so that's the different colors here. So Stack Server, first of all, well, that's an API server, right? You set it up for your static catalog, for example, ingested, and then it just gives you API capabilities, like searching, aggregation, et cetera.
01:01
That's great. And then, also in the Node ecosystem, we have the Node validator, so that you can use as a command line interface for validating your Stack catalogs, basically. And then there's all this, like, browser-based libraries here, so Stack.js is a recent addition,
01:23
which is basically, if you have deserialized from JSON into a JavaScript object, a Stack item or a Stack collection, then you usually access things like, you do, I don't know, search for links, right? You do object.links.filter,
01:41
and then get specific filters, right? And what it adds, basically, on top, is different methods that you can use, so that you don't need to write all the code for accessing these properties and links, et cetera, so there are easy ways for doing that. I'll go into that later. Stack.js gives you TypeScript bindings for the Stack.json,
02:01
so for the Stack specification itself, not for Stack.js, it's not, no code, it's just really the bindings for the, yeah, the JSON part, basically. Stack.js client is not there yet, but that's an idea to build on top of Stack.js. It's basically API component, and then there is also Stack Helper right now, but that's unmaintained, which does the same job.
02:22
Maybe we can merge these together. There are Stack layer and open layer, Stack implementations where you can just throw in Stack, and then it gives you all the information from Stack as a web map, basically, so it gives you the bounding boxes, shows default, if it's there, or thumbnail on the map.
02:42
Stack.migrate is to upgrade your Stack.json if you still have an old version of it, in 0.9 or 0.6, it goes back to 0.6, into the latest version, that's the command line tool interface as well. Stack fields, open your view components for metadata rendering in HTML. Stack browsers and web UI, which we'll talk about later as well.
03:01
And Stack validator, that's also an idea. I mean, we have all the validation code in Stack node validator, so we just need to, at some point, just grab all the things that are in there and just make them available as pure JavaScript, basically. So that's basically the ideas, and that's how it should all work together. The more light things here are the things
03:23
that are still in work in progress. All others are already there. So you see there's all interconnected ecosystem, basically, building on top of Stack.js now. Stack.js adds the bindings, basically. JavaScript clients, that's the API functionality. Stack.layer and stack.ol.stack basically use Stack.js
03:41
for making the web maps available. Stack.migrate is used to migrate older versions, if you read them in, and they are old version, to have a uniform interface into your application, it migrates to the latest version. The validation part is here. Stack.browser uses a lot of that to make a web UI. And then Stack.fields and OpenUu components are for HTML rendering, basically.
04:01
So that gives them a good overview of how they all depend on each other. Yeah, so Stack.js, here is an example at the right and what it does at the left. So if you have a Stack.js and basically you import this, you can create an object from it, you pass it in, and then you still have the direct property access
04:23
as before, so it's really a drop-in. You just need to add a step in between, and then it gives you all the helper functions on top, but all your other code still works because it still offers all the property access, basically, that we had before to the ID, for example. So you can still do object.ID as before, or object.Properties, et cetera. But you can also call these methods there
04:43
to check whether it's an item, to get some names, to get a default geotiff based on the ranking that you can customize. You can get assets with a specific role easily, or get a root link, et cetera. It's all these convenience methods that make it work in JavaScript, but just on the JSON objects, a bit tedious. And it gives you a nice class-oriented interface
05:03
that you can see on the right side, it's small. It's all the classes and how they belong together. Yeah, so that's there. You have this, or this is used as base for other packages, of course. So if you use all the other things, you get basically Stack.js objects directly,
05:23
which you can use as well if you use the other libraries. And it automatically uses, for example, also Stack Migrate, that if you create it, it automatically migrates to the latest version so that you have a stable interface, basically, to access. Stack.js client, well, that's still an idea
05:41
not implemented yet. There are Stack Helper, which is unmaintained. We hope to merge that together so that it uses Stack.js in the end, and then gives you, basically, what in the Python world is PyStack and PyStack client. Here, it's just Stack.js client, and then PyStack client would be the equivalent. It gives you the HTTP access, basically,
06:00
to APIs that you can filter, that you can request without worrying about how the endpoints are called, et cetera, just to give you a nice interface, basically, as a CQL builder. It's meant for the API on top of that so that you can do HTTP requests, and that would also go into Stack browser in the end later.
06:21
If you're interested in that, please talk to me later, and then you can figure out whether there is anything that we can do together, maybe. We have Stack Layer, and that is for Leaflet, and then there is also a PR for OpenLayers that adds the same functionality with a very similar interface, where you just add, basically, a layer
06:41
into OpenLayers or into a Stack Layer, and then, by default, it shows the geometries, or bounding boxes, from the item or collection that you passed in, checks whether it goes above the end to meridian, makes the meridian aware so that you don't get these weird bounding boxes over the end to meridian,
07:01
and then you can actually also show cogs or thumbnails or normal geotiffs if they are small enough, and then, based on the metadata that is in the Stack Asset definition, it does nice visualizations, right? If you have a cog with floating point numbers in there, by default, you usually don't get nice imagery
07:21
because it's all black and large scale of numbers, or you have statistics in your Stack item, then it knows which is the minimum and maximum value, or maybe there is a histogram or a classification extension in there in the metadata, and then it automatically derives from all the extensions that are out there, and it gives us this information,
07:41
and tries to get the best, basically, visualization of that cog. It's all client-side, or can work all client-side based on geotiff.js, but if you don't want to use that or have a tiler anywhere running, or there are cogs that are not supported by geotiff.js, it can also fall back to using a map tiler
08:02
like tie-tyler, for example, and then just request the ties from there. It can also do basic bandwidth, both libraries can do basic bandwidth, and you can also pass in the lists and lists of collections and items that you get from an API, for example, and then it gives you a list,
08:21
or a map with multiple boxes, basically, that you can click on, and then you can select a specific item based on top of a map view as well, so that you get all, just by passing in the JSON, basically. Stack fields is more specific to rendering in HTML in the browser.
08:41
It basically defines rendering hits for stack fields that are not in the schema, right? So, for example, the label in a specific language, or it gives an explanation about, like if you have, for example, here, GSD. Oh, that mouse doesn't work here. Anyway, if you have there the GSD, right, there is, you can't see it here,
09:03
you can hover over it, and it gives you an explanation for what the GSD is, basically, and that all renders nicely in HTML that is used, for example, in stack browser for the rendering of the metadata fields, and, for example, also, PyStack uses the fields JSON file that has all the specification in there
09:21
to generate summaries, for example, so that's all about enriching the JSON schemas of stack with additional metadata that is used for rendering or generating other things. And the library here has a lot of helper functions, for example, to format license, right, you pass in the stack JSON,
09:40
and then it looks at the license property, what's in there. If that's very proprietary, for example, then you need to look at the link section. The license link there has probably a title, and then it generates nice HTML link, basically, with the title and the link. So that's all done for you. It renders individual fields by HTML in the end,
10:03
and groups them by extension, if required. So, as you see right here, right? OpenView components is from the OpenView ecosystem, basically, but does also things for stack, so you can just render, for example, list of data collections, as you can see here, or list of items, for example, easily,
10:22
so then you don't need to use all the specific low-level things from stack fields, but it gives you a nice interface as that component, so you don't even need to use JavaScript here. You can just embed an HTML tag with a JavaScript link to your HTML document, and that embeds it easily. You just need the HTML tag, the JSON in there,
10:42
and then you can use it in static HTML, actually. And then it gives you nice views, you like collection views, et cetera, I hope that was not intentional. You can also render individual collection or individual items in context where you don't want to use JavaScript.
11:01
For validation, there is stack node validator right now. That's in location via command line interface. Easily, if you have node installed, just use npx stack node validator in the file or folder, and it validates everything in there. You see how it looks like in there. It also checks the extension, whether it's valid against the extension schema. It can lint and format as well a bit, just a bit.
11:23
There's no real best practice checking, more like whether the file looks good, whether the JSON is valid, et cetera. Can be used for mass validation. It accesses HTTP and file systems, so you can also access, just give a URL here, for example,
11:41
and just linting and formatting. What I want to do still is to extract the validation part to a separate JavaScript library. It's not much work, just basically extracting out the code, and never got to it yet, so that you can invocate it from code. For example, I could use stack validator then in stack browser to validate and give you, actually browsing through the catalog,
12:01
a nice list of things that may not be compliant to the specification. Then you also have stack browser. There's a dedicated talk about that today in this room here again at 16 o'clock, where I show all the new features of stack browser
12:20
with nice visualizations, et cetera. So it got redesigned, it got a lot of new features in there, just maybe attend the session there. And yeah, that's a web user interface for static stack and stack APIs, basically. It can just show there it's HTML validation, JavaScript, and CSS files,
12:40
and then point it towards a specific instance, and it just works out of the box. Stack Server is the API interface. It's basically based on AWS. Gives you easy deployment on AWS. It supports open search. This is not the OVC open search, but the AWS clone of Elasticsearch, basically,
13:04
works on top of that. It's serverless, based on Lambda functions, basically supports aggregation on the core API interface, basically, what it doesn't support right now is a filter extension, which is a bit of a bummer, but that's mostly due to that there is no CQL library out there in JavaScript yet that can parse CQL, basically.
13:26
I didn't see a five-minute sign yet, so I guess I can still go into the excursion here. So that is from last year, basically, where we gave a longer talk about how to use cloud-native geospatial file formats in JavaScript.
13:42
That might be also interesting because if you're in JavaScript anyway and reading stack, you probably also want to do something with the assets and the files that are linked to. So for discovery, of course, we have stack for us, and array data, there are different file formats like Coq, PMTiles, Kotar, ZO-ZIP, and ZAR that you can use, and for vector data,
14:01
there's Geopacave, Geobuff, and Copic. Just for some of them, there are JavaScript tools out there at the end. I'll have a table that gives an overview, but let's just dig into some of the specifics here. For example, for geotiffs in general, and especially cloud-optimized geotiffs, there is reading capabilities and various variants.
14:21
There is GeotiffJS, which is a pure JavaScript implementation. Low-level reader of most Cocks. Some of them are not supported, but most of them are. If you need more or something that is more speedy, you probably want to go into GDAL JS, which is Wasm-based, WebAssembly-based, and Loam, for example, gives a nice simple interface
14:42
on top of GDAL JS. There's also GDAL 3JS. It's a newer Emscript part of the GDAL into WebAssembly as well, so one of them in WebAssembly is available for you. For visualization, for example, there's GeoRusterLayer for Leaflet, which StackLayer also is built on top of.
15:01
That's for web mapping with good defaults. For Leaflet, basically, that's just the bounding box rendering you get and all the cock rendering, and there's also open layers, which is also for web mapping, of course, but this time, it uses WebGL, which, of course, makes it much faster to render,
15:22
but the implementation right now is still less automatic for cogs, so you need to give them more details, basically, when displaying the map up front, so it's not as automatic as GeoRuster for layer for Leaflet, which actually goes into the data and derives statistics for it so that it nicely visualizes the cogs.
15:44
For analysis, there's, for example, GeoBlaze, which allows you to work on cogs and, for example, compute summaries of areas, so sort of statistics, or does bad math for cogs, so that's also out there. For cloud-optimized pound clouds,
16:01
there is one implementation, which is called CopicJS, which is written in TypeScript, but it depends on last perf, so that is wasm-based, again, so you would, if you're more like pure JavaScript, then, well, there is nothing for you because it depends on wasm in the end.
16:23
The issue here is that the code is out there, but there is very limited documentation, so you would need to dig into the Copic code, actually, to understand it, so that is a bit of an issue because, well, the alternative, of course, is to contract the maintainer, of course, so that's a bit of an issue in this regard right now.
16:43
ZAR, and especially the evolving GeoZAR, has no specific things for GeoZAR right now, just an open layers prototype, as far as I know, at least, but there are at least, for reading capabilities, there's jar.js, and there's zar-js,
17:02
so small differences to be aware of here. ZAR-js is more full-fledged for reading and visualizing ZAR-formatted results, and zar-js is a bit lighter, minimal alternative if zar.js is too heavyweight for your use case.
17:22
For realizations, there is carbon-plan maps, which is a relatively fast rendering WebGL-based implementation based on top of one of these, I think, on top of zar-js. For geo-parquet, again, there are libraries for parquet, but now specific libraries for geo-parquet,
17:41
but geo-parquet is just a bit of JSON encoding on top, so I guess you can make that easily implemented yourself. Here, the issue again is that the pure JavaScript implementations are both abandoned because they are implemented in parquet, and JavaScript is relatively tricky due to the binary format and all the compression things
18:02
that are going on there internally, but there are two VASM-based implementations, GPQ based on Go, or parquet VASM based on Rust, so one of these you could use if you're into WebAssembly, and loadas-gl also supports this based on one of the VASM implementation.
18:23
For flatgeobuff, reading is supported, writing limited support, it can't write index data. It's supported in loadas-gl, and for the mapping libraries, there's also some implementation or code snippets out there for lead and flit.js and OpenLayers. It's also a plugin for mapbox-gl,
18:41
which not only works for mapbox-gl, but also for map-lever-gl. And here you can see an overview, which doesn't quite work in PowerPoint, it seems, converted from Google Docs, so better use the slides later that I've linked to, rather than this, which doesn't fully work.
19:00
Anyway, so for Coq, there is good JavaScript support, I would say, there is a library out there, you can use visualization tools, it's good usable in browsers, and documentation examples are good. For Copic, as I said, it's a bit limited, there is a library out there, but everything else lacking right now. Flatgeobuff is pretty good usable as well in a browser,
19:23
but be aware there are no overviews in Flatgeobuff, so you need to read too much data in some cases, that might be difficult. In Geopaque, there is VASM again, the geo part is missing, so depending on what you want to use,
19:41
it's either good or bad, depends on the use case. For PMTiles, there is good support in open-source software. For Stack, there is good support, as we've seen in this talk. For GeoZAR, there is at least ZAR libraries, but depending on the use case, ZAR in the browser is also difficult. If the chunk size is very large, it's difficult in the browser due to the memory constraints
20:02
so that's a bit of a tricky thing. And that's it. Thanks for your attention.
20:20
Thank you, Maggios. We'll start right here, so if we have someone to feed the bugs, the bot is nice to point to the bot,
20:41
he knows how to feed the building. Yeah, do you have any idea of all the time? In the browser, no. There's just Copic.js, which is WebAssembly-based, and there's just the code. There's a GitHub repository with the code, but there is literally just one example and no documentation at all of the API,
21:01
so that's pretty tricky to get in. I don't know of any alternative out there, so that is right now the state we have for Copic. Thanks for the great overview.
21:23
For vector and stack, is there a special mechanism for people to do that, I mean, for example, you have a vector that actually was via a stack search. Yes, people do that, especially I've seen recently also
21:41
geo parquet, for example, in often, what is done, for example, in planetary computer is that they have all the stack items in the geo parquet file as well and offer that from collection assets, basically. But I've also seen geo parquet files and shared flat geo buffs in item assets. And then there is no specific best practice,
22:02
I mean, there is a table extension, for example, that is relevant there, probably, but otherwise, they just use normal ink, media type, and whatever you use for all the others, right? But there is no specific vector extension rather than, except for the table extension, right?
22:20
Yeah. Okay, we are here.
22:43
All right. I'll take a question. I wanna ask a little bit about this ecosystem. So, XAR, XAR dumps, or XAR packets, or something,
23:01
are they also accessed here in the extent, or can they be exposed to the system in XAR? That's a tricky question. I mean, XAR itself is a file system, has different folders and files, and the file system, right? So, I haven't done that yet. We've talked a bit with the community, but we couldn't really figure out yet how to do it,
23:23
or are we aware of more of that? Yeah, it's tricky, because XAR actually does a similar thing in stack. It has it. And so, if your XAR files are, like,
23:41
is this typical of homogeneous data or model data, then using stack might actually not really make all of sense, because you just have, essentially, one data file. Like, maybe that could be a collection, or something like that. However, if you have XAR tables, like, oh, I have, like, several data you could mention
24:00
for the XAR that you've seen, then I think stack makes more sense for that purpose. I mean, nothing forbids to link to the XAR archives, right? That's, of course. So, one point to look at, like, general catalog that then points to the project. Yeah, exactly.
24:22
You would want to use stack for XAR in spatial circuits, but if you just have one that's our dataset, then they still handle that spatial, search it on kind of a list when you do something. Any other questions?
24:44
You have my email address. Oh, go ahead, one more. Is there something like your asset? I mean, I'm thinking more like in the small scale, not for, like, hundreds of satellite data, but for, like, a small project,
25:01
maybe, like, a scientific project, where you have lots of different little things like this, one vector file that shows, I mean, capturing areas, and then some data sets, and then you would rather want to do that, maybe, to edit, like, visually, just because you don't need it. Not that I'm aware of, but that would be something that
25:22
I would look forward to have in stack browser at some point. But, Michelle, no, not here anymore. I think someone is working on that, but I don't have the details. Okay, without further ado, we're out of time.