We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Paituli STAC - experiences of setting up and using own STAC catalog

00:00

Formal Metadata

Title
Paituli STAC - experiences of setting up and using own STAC catalog
Title of Series
Number of Parts
156
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
In this presentation we discuss our experiences from setting up Paituli STAC, which contains open Finnish raster datasets. We did not do any software development, but decided to use GeoServer with STAC extension. Own code was only written for populating the PostGIS database with information about ~100 collections and ~250 000 items. Paituli STAC catalog is mainly targeted for data analysis use cases, but can be used also from web applications. We have also prepared public example scripts for using Paituli STAC with Python and R. We will also show the results of some scaling tests of using data from STAC on a supercomputer with Dask and xarray.
Keywords
127
Streaming mediaLibrary catalog
Data analysisService (economics)Computer configurationRaster graphicsVector spaceImage registrationTrigonometric functionsSupercomputerDatabaseLink (knot theory)StatisticsWeb browserWeb pageScripting languageService (economics)Online helpLibrary catalogServer (computing)Fluid staticsMusical ensemblePoint (geometry)Flow separationComputer configurationSoftwareMultiplication signLatent heatLibrary (computing)Open setRight angleData analysisSlide ruleCASE <Informatik>File formatProjective planeModule (mathematics)Field (computer science)MetadataCombinational logicMoment (mathematics)Template (C++)Set (mathematics)BitParallel portDefault (computer science)CurveProcess (computing)Raster graphicsVector space
Least squaresComputer-assisted translationComputer animation
Transcript: English(auto-generated)
I'm Giliak and I'm from a Finnish company CSC and we have put up a stack catalogue with Finnish data and the starting point was the data was available already but it was in many places and it was difficult to find. So basically in this project we were only concerned about putting up the stack and I put out
also here the time that things have taken because it took a little bit longer than we thought in the beginning. So in total we're talking about one and half years and we currently have only raster data but of course we have also lighter and vector data so thinking about that and all data is open to
everybody and we have used only phosphor G and if anybody is actually interested in a service then there is a link. The first question we had because there is actually several options to choose when you want to put up a stack API then we spent quite some time in selecting which
software to use because we had very clear that we are not going to make another one and we ended up with gear server with open search for AR community module and it has post case as the back end and for us the main reason maybe to choose this combination was that we had both in use
already so didn't get really new dependencies and that's kind of nice and also we were like familiar with both gear server and post case so also the learning curve was quite easy there and one thing that I especially liked was that both database and templates were super easy to modify so if we wanted
to add some field or something then that was very very easy and it also had an API for making data updates because some of the datasets are updating daily so that was also important thing that to keep it up to date in the future and one
thing that we have that is our internal thing that we have scripts for making gear server statistics so now I could get all the stack statistics out what we did not like that much was that there is no tool to ingest such static stack catalogs to gear server we have kind of written it now ourselves but it's not yet good but maybe someday we could have something
for that to to provide for others and then one problem for again for us was that we don't have Java coders in house so whenever there is a problem then now we have a solution for that that we can talk to care solutions and in some cases it would be nicer for us if if the chase them would be
default format now it's HTML I think I skipped on the previous slide that our main use case are data analysts from academia who want to actually use the data from Python and R so that HTML doesn't make any sense at the moment we
are talking about hundred collections with 350 items and this has been the main work to get the metadata to the database it's over 12 months I would say of course there are nice Python libraries that we have been using but just to get everything right it takes surprisingly a lot of time we have put
our scripts out but they are very data-specific so I don't know if they are for any help for anybody else one very nice thing was that browser here we didn't do anything we just linked our our stack and it it was there and
this is actually when there is new users this is the page where I refer that go there check what it is available and then we have statistics I thought this might be interesting so it's mostly Python users if you are an FME users but I guess this is quite expected because I personally like the
Python tools a lot more and then we have also written example script for Python R the Python one is quite advanced with Dask and parallelization in the background because as a company we have a lot of computing resources that we provide for Finnish researchers so this was basically what
the end users are interesting and then as the last thing we have tested to run this Python thing in parallel and it kind of evened out after five parallel processes so if anybody has any good hints how to make this curve go longer
than I'm very interested yes