News from actinia - let's STAC!
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 351 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/69200 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Vector spacePlug-in (computing)Core dumpAsynchronous Transfer ModePresentation of a groupRevision controlOpen setPhysical systemFluid staticsComputer fileBitPlug-in (computing)INTEGRALArithmetic meanGrass (card game)Planning2 (number)Device driverComputer animation
01:43
MereologyINTEGRALAuthorizationProjective planeGeometryComputer animation
02:14
Asynchronous Transfer ModeMechatronicsDevice driverParallel portPlug-in (computing)Vector spaceCore dumpPreprocessorGrass (card game)Point cloudGrass (card game)Representational state transferCodeUniverse (mathematics)Scripting languageMereologyComputer animation
02:58
Grass (card game)Projective planeSet (mathematics)MereologyModule (mathematics)Physical systemUniform resource locatorLevel (video gaming)Directory serviceModule (mathematics)Grass (card game)Image processingFeldrechnerProcess (computing)
03:44
Representational state transferData managementGrass (card game)Electronic mailing listVector spaceModul <Datentyp>Group actionLimit (category theory)Raster graphicsTemporal logicThermodynamischer ProzessCalculationSet (mathematics)PermanentComputer-aided designData storage deviceType theoryResultantLink (knot theory)Line (geometry)Integrated development environmentCASE <Informatik>Set (mathematics)VolumenvisualisierungPhysical systemRight angleAddress spaceNeuroinformatikChainProcess (computing)Task (computing)Latent heatGame controllerMultiplication signEntire functionDifferent (Kate Ryan album)Functional (mathematics)Point cloudOrder (biology)NumberGrass (card game)Electronic mailing listoutputThermodynamischer ProzessScripting languageModule (mathematics)Server (computing)Representational state transferData centerTime seriesRaster graphicsCore dumpOpen setContent (media)Instance (computer science)Table (information)Software testingBitLevel (video gaming)String (computer science)LaptopComputer configurationClient (computing)Speech synthesisHierarchyProjective planeDirectory serviceMappingView (database)Uniform resource locatorData conversionVector spaceReal numberDrop (liquid)Query languageOpen sourceSoftware developerComputer animation
09:52
Router (computing)Plug-in (computing)Pole (complex analysis)Vector spaceTube (container)Core dumpFile formatShape (magazine)MultiplicationContent (media)Data typeDefault (computer science)Server (computing)Computer configurationLocal ringSynchronizationSet (mathematics)Dependent and independent variablesReal numberSystem of linear equationsAreaVoltmeterEvent horizonoutputProcess (computing)Virtual realitySpacetimeInstance (computer science)Raster graphicsCubeJava appletScripting languageClient (computing)Point cloudFunction (mathematics)Open setData managementElectronic mailing listThermodynamischer ProzessStapeldateiComplete metric spaceTheory of everythingImplementationGrass (card game)Device driverText editorUser-defined functionData modelCodeRun time (program lifecycle phase)Computer fileLattice (order)PiComputer-generated imageryOrdinary differential equationTrigonometric functionsService (economics)FreewareOctahedronDecision tree learningRepository (publishing)Module (mathematics)TesselationParallel portView (database)AuthenticationModul <Datentyp>Type theoryThermodynamischer ProzessPlug-in (computing)Instance (computer science)Open setText editorClient (computing)Core dumpShape (magazine)Computer fileFile formatPresentation of a groupProcess (computing)Remote administrationFront and back endsKey (cryptography)MathematicsWeb 2.0ChainWindowFunctional (mathematics)Different (Kate Ryan album)Revision controlMereologyInformationEnterprise architectureMultiplication signLine (geometry)Slide ruleInstallation artLibrary (computing)outputParallel portQueue (abstract data type)Physical systemProjective planeSet (mathematics)Grass (card game)Device driverPoint cloudPoint (geometry)BitTesselationAuthenticationComplex (psychology)Code refactoringAreaMappingINTEGRALVector spaceValidity (statistics)Module (mathematics)Software developerMusical ensembleRight angleFlow separationArithmetic progressionGoodness of fitSpacetimeScripting languageStandard deviationGeometryStack (abstract data type)ResultantTelecommunicationComputer configurationDatabasePixelLink (knot theory)Representational state transferAsynchronous Transfer ModeRaster graphicsResampling (statistics)Configuration spaceComputer animationProgram flowchart
Transcript: English(auto-generated)
00:00
Yeah, thank you very much. Thanks for coming As you can see, I'm not Carmen Tavalica who was supposed to give the talk. Unfortunately, she fell ill This night and she is not here. I mean she's in Florence Greetings to you, but she's unable to deliver the talk. So it's a
00:21
Micro surprise for me that I'm giving the talk now, but let's say new to everybody. So that's fine second thing was that the So maybe some of you have been in Bucharest at the conference. She used this newspaper style so where you move around in the newspaper and this was exactly the plan, but
00:41
The HTML was eaten by the system. And so I had to convert it to static PNG files never mind. So imagine it would smoothly fly around Yeah, whatever. So the topic is we are presenting news in actinia What has happened in the past 12 months and there's been happening quite a bit
01:02
Overall let's say you can see what's here So we want to give short notes about actinia itself in this presentation Additionally the latest features of course, which have been implemented. We are now meanwhile is at actinia version 4 so
01:20
this is Comparable faster than the major revisions of grass GIS, for example Where it took us almost 40 years to reach version 8 Anyway, so then I will talk about actinias tech plug-in open EO API Open EO grass GIS driver. So let's say lots of integration. We are looking at
01:45
We are part of Mundialis Actinia is importantly an OS geo community project. It's not a Mundialis project I mean we are maybe the main authors original author was during gabbard back then
02:01
But we are taking care and of course, it's on github and everybody's invited to contribute In the corner you see Jorge he was being developing the actinias tech integration you are doing this as a student from University of Munster for being in
02:21
in Bonn for a while remotely at least and He contributed this part. So next chapter is introduction to actinia who hasn't seen anything yet just to give you an idea and actinia is a REST API around grass GIS, but not only we also have support for
02:42
is a snap for example or GDAL is naturally part of everything so you can also bring your own code like Python scripts or executables Whatever you want. So whatever can be automated can be put into the workflows There are two important concepts Some of them have seen this but I'm very fast on this just to understand what we are talking about
03:04
Grass has this notion of locations and map sets You can imagine this like project directory with subdirectory just to better organize the data and importantly one location or project must have one projection only And then the other part is that grass is a modular system with plenty of modules
03:26
Doing particular things they are organized in a natural way like I dot something is the imagery image processing V dot something vector processing and so on and the second part of the Name suggests what the module does. So like this we have up to
03:42
500 of them so quite a few and This is of course something which which you might want to orchestrate in a cloud environment So why having this wealth of different functionality and why not? putting some clouds cloudish stuff around like a REST API in order to
04:03
develop workflows and to major number crunching using this technology So we put a REST API on top. You can naturally list all the data content and so on You can also control on a user specific base which
04:22
Commands a user may use and may not you have the natural the common user access control and everything also limits for pixels so that if you have Let's say cloud instance Which you have to pay for which is normal that your user doesn't burn your cloud budget in no time by just processing
04:42
Entire continents or something like that so you can limit things here limit the duration of the computation and so forth Yeah, this is possibly enough So how to deploy everything? We have different options here Those familiar with Docker and Kubernetes. They may already know what I would say. So you you can imagine that
05:07
You have small actinia workers which are floating around in the cloud and those are then orchestrated by the the core system and Well, you basically launch these workers if you have a job and then it depends on your data size if you want to split your data and send them
05:25
To different nodes or not. This really depends on the problem. For example for time series you could If you have a stack of data must not be time series can even be something else You can send them to different nodes and at the very end you reassemble all the results
05:43
We have different technologies here where we can deploy stuff on this depends some customers We are using this also in in our activities They have their own cloud and say you need to to launch things on on our infrastructure Which is possibly protected and so on and since it is
06:02
Open source, there is no problem with that Docker swarm kubernetes so we can even use mini pot I don't know if anyone use knows that you can run it on your laptop and basically launch your Micro cloud instance with everything on your laptop if you want to play around and pretend to be a big data center
06:22
For example in order to test workflows on tiny data sets and then you eventually deploy it in the in the real data center Okay now a bit of rest API, how does it look like if you want to Query the system what data are on the server or in the infrastructure?
06:41
Let's say you can send a rest put get request slash locations will give you the list of what is there So here we have it's a bit small. I apologize, but this is also related to this conversion. I Try to tell you what is that? so on the left side get locations and then you get the names of those which are projects let's say and
07:03
On the right side you can look into the sub directories and see what data sets are sitting inside Yeah, this would give you in not really visible a list of maps And additionally if you want to get a quick view what map is there and how does it look like you can just use
07:24
The render in endpoint. So one of those things is called endpoint in rest Speech rest API and You can see it's a kind of hierarchical system which you Where you send the request to the to the server and something comes back and in the lower line case there
07:44
You say I want to see this data set this particular name slash Render and then it will send back the picture so you can take a look this of course comes in JSON encoded so you need some client for this, but there are different clients available
08:01
To do things like that. Well, so then we have different types of processing user-defined processing the so-called if you mirror and the persistent Ways of doing things persistent as the name suggests the data rest on the server You would have to pay for the storage if you are in a paid cloud environment, of course
08:20
So you need to think about it twice and the if a mirror stuff is you want to compute something or the? You don't care what happens to them in future. So there will be automatically trashed and The result you are offered a download link for example, or it is sent to a cloud-based storage or whatever you prefer
08:40
And there you can then go and see sorry consume the results So now how to do Different process chains So we call it process chains what used to be called maybe a script or a job or something like this So it's essentially a list of tasks to be done
09:01
They are encoded you see it on the right side in JSON So you say step one step two step three and by the way the input to step one is this data set It can be even sit on a remote System you'll give HTTPS some address and it will fetch it from there
09:21
and Then you do the processing you say what you want to compute and eventually you will get the result and Already mentioned we have support for all the grass modules We get meanwhile also the explanations what the module does You can retrieve automatically from the system. You can also ask it. What are the input data? What are the input types? Is it?
09:42
Vector map a raster map a string a table whatsoever and this can be queried automatically Yeah, the rest I have already mentioned okay back to the newspaper and to the next topic so Now I will see the classified so small news what has happened in the past and zooming to them
10:07
What we added in the last month Let's say so vector upload is now there by an endpoint. You can upload data in different formats Geopackage here Jason and good old shapefile as well
10:21
Seems not to disappear for the time being And here's an example. This is maybe not really useful now, by the way This the slides are on github so you can watch them in full quality and even click on the links then Just as a hint so what else have we done enable a separate Redis queue so Redis is a database
10:44
Which is managing the different jobs imagine you have the cloud instance the users are happily sending stuff to the cloud And then it needs to be distributed to the different worker nodes and results connected This maybe didn't come to an end this one has ended and please notify the user and whatever communication you can imagine
11:03
So the actinio instance receives jobs and writes it to this Redis that's a database type queue and then somebody is looking at it of course the system and Doing one after the other and possibly you can even use priorities or whatsoever
11:22
So sometimes it may be useful to have different queues available and this support we have been adding Well and some other specific technical stuff which I will skip for now and Starting the worker is nothing more than these few unreadable lines where we say Start custom worker queue name and the configuration and then the thing runs
11:46
Okay, then very nice very interesting a new actinio Python client This is maybe quite interesting for some of you So you can now write a few lines of Python to remote control the actinio instance you say import from
12:01
Actinio, and I think it's not in the slides. Maybe it comes. I don't remember You can say even pip install actinio core, and so you get through pipey you can now install actinio itself easily okay, so here we have the access to
12:21
Use actinio through Python, and we will also bring other clients. I show later Likely to use this Python library because it's quite convenient you see the functionality Get version especially you can authenticate get information about all the data You can delete stuff if you have something which belongs to your user space which you want to remove
12:44
And you can do also process chain validation imagine you write this Jason's Jason scripts with the different process steps to be done Then you need to validate it if it's syntactically Syntactically correct or not and there's an endpoint for this which tells you oops. There's something wrong here, please fix
13:05
Is by the way how to write these process chains you can use grass itself Whatever command minus minus Jason will show you it show the thing in Python In Jason of course and then you can put it to your
13:20
Process chain and then put one together we have also clients To do it differently, but I will come to this Okay What else we have a tiling plug-in and actinio parallel plug-in so from plug-in name you can already Imagine that we try to so we did some refactoring we try to modularize actinio
13:43
them as much possible as As much as possible so the tiling thing is separate the Parallel thing is also separate now if you need it you can install it otherwise no need the idea of tiling is to to chop the area and especially raster maps of course into pieces and then send the
14:03
individual tiles to different nodes for easier parallel processing you can define how the tiles should be Configured it is written there so width and height and so on and the rest is done by this plug-in The parallel mode plug-in is still work in progress
14:22
Here the ideas to take one process chain and split it into pieces so perhaps The second one part doesn't have to have the previous one But you can send it in parallel do several steps in parallel and then re aggregate it of course you have to have Kind of stop marks in the process chain itself to tell the system what can be split in what cannot
14:47
Imagine you want to do segmentation and you do segmentation then on different input bands Why not do this in parallel and then reunite everything for the subsequent classification for example all right?
15:03
Yeah More features I will not read them now Especially the refactoring is interesting and yeah lots of small things with pipelines with Resampling options if you have had once you have heterogeneous data-sets you need to
15:22
To fix the vector raster geometry the pixel geometry and align them somehow in a clever way and all this kind of stuff Is there? Okay next topic would be a tinya stack plug-in that is fairly new developed mainly by hawker I already mentioned that and
15:41
yes, and Here we have differ. I don't want to go into stack in detail because this has been already presented, and there's no time anyway Remember stack plug-in is there. I need to speed up so we have different create read delete and so on things Collections and instances what is offered in stack usually then we have a second porter
16:06
naturally a stack exporter so you can get data in and out and This would be the sec part Let's talk later if you have more questions Then where all was connected years ago is the open EO API So we continue to develop a tinya to be compliant with open EO and here open EO is an API for
16:28
standardization of Let's say cloud backend so that you can use Open EO API and in the back end that could be this one or that one or actinia and This is what has been a European project, which is also still under
16:44
development different endpoints as usual We married of course open EO and stack as well, and this was the open EO part How does it work We use a driver for this open EO grass GIS driver
17:02
this has been developed a time ago, and we continue to develop this one to stay compliant also with Future changes in open EO and to bring things together for users There's just to tell you and a nice web editor existing in open EO which gives you the possibility to
17:21
Graphically put together a workflow and then Use it later on stack again also here and Here you see a process chain the different small windows No, sorry these small items which are connected and you can graphically arrange them and then send the job to
17:45
To the back end, okay, I come almost to the conclusion the outlook We are working on kick log integration so that is user authentication And this will take some more time because it's complex stuff
18:01
We want to restructure the modules thematically because with 500 or what Different function functions, it's a bit of an effort to Understand what's to be used, but this since the modules are able to auto declare themselves this should be a durable thing we work continuously on the Python client and
18:25
We'll continue as I mentioned with the parallel plug-in so everything is sitting on github a bunch of plugins with actinia core and actinia API To study the API itself all the plugins and different clients. We also develop QGIS
18:44
plugin and This will be published at some point when it is usable, let's say and You are importers exporters and hem charts if you want to deploy it yourself And this would be my presentation, and I thank you for your attention