GRASS GIS in the cloud: actinia geoprocessing
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43301 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Bucharest 2019282 / 295
15
20
28
32
37
38
39
40
41
42
43
44
46
48
52
54
57
69
72
75
83
85
87
88
101
103
105
106
108
111
114
119
122
123
126
129
130
131
132
137
139
140
141
142
143
144
147
148
149
155
157
159
163
166
170
171
179
189
191
192
193
194
195
196
197
202
207
212
213
214
215
216
231
235
251
252
263
287
00:00
Group actionProcess (computing)Point cloudLecture/ConferenceMeeting/Interview
00:17
Grass (card game)Point cloudSoftware developerSoftwareProcess (computing)Social classPoint cloudCASE <Informatik>Multiplication signSet (mathematics)Computer animation
00:41
DialectExecution unitWeb browserDifferent (Kate Ryan album)WebdesignWeb 2.0Building
01:15
Grass (card game)Functional (mathematics)File formatTesselationProcess (computing)ChainDifferent (Kate Ryan album)InternetworkingType theoryRepresentational state transferWebsiteService (economics)SpacetimeProgramming paradigmInternet forumContext awarenessLevel (video gaming)GeometrySet (mathematics)BuildingBitUniform resource locatorLipschitz-StetigkeitAlgorithmProjective planePoint cloud1 (number)Software developerModule (mathematics)Vector spaceCodeRaster graphicsExploit (computer security)Functional (mathematics)Panel painting
03:43
Representational state transferElectronic mailing listModul <Datentyp>Grass (card game)Data managementGroup actionLimit (category theory)Temporal logicRaster graphicsProcess (computing)CalculationFile systemUniform resource locatorScaling (geometry)Physical systemRepresentational state transferRead-only memoryProcess (computing)Data managementMehrplatzsystemSocial classSet (mathematics)BitPoint cloudReading (process)Office suiteComputer animation
05:11
Different (Kate Ryan album)Asynchronous Transfer ModeDatabaseMultilaterationRepresentational state transferProcess (computing)CalculationState of matterPhysical systemMultiplication signComputer animation
05:49
Grass (card game)PermanentCodeParameter (computer programming)Computer-aided designUniform resource locatorTelecommunicationDependent and independent variablesProcess (computing)LastteilungFunctional (mathematics)Physical systemDifferent (Kate Ryan album)Connected spaceDatabaseRaster graphicsConfiguration spaceBitVolumenvisualisierungRepresentational state transferSoftwareElectronic mailing listMappingChemical equationPoint cloudLevel (video gaming)DampingShared memoryComputer clusterInformationUltraviolet photoelectron spectroscopyDiagram
08:09
Process (computing)ChainMusical ensemblePoint cloudAlgorithmProcess (computing)ChainGoogolEndliche ModelltheorieData storage deviceComputer configurationType theoryPoint (geometry)DemosceneAlgorithmResultantComputing platformPoint cloudUniform resource locatorDifferent (Kate Ryan album)Electronic mailing listSystem callCalculationFunction (mathematics)ImplementationRight angleLaptopMultiplication signLevel (video gaming)Module (mathematics)File formatPlug-in (computing)Generic programmingDatabaseWeb 2.0Computer animation
11:14
AlgorithmProcess (computing)Computing platformData storage devicePoint cloudMusical ensembleUniform resource locatorGrass (card game)Graph (mathematics)AlgebraProof theoryCore dumpInterface (computing)System callCalculationUniform resource locatorProcess (computing)DemosceneDependent and independent variablesResultantChainEngineering drawing
11:58
Graph (mathematics)Grass (card game)AlgebraParalleler AlgorithmusInterface (computing)System callGraph (mathematics)Revision control2 (number)OntologyInstance (computer science)Open setMultiplication signProjective planeThermal conductivityFormal languageOpen sourceProcess (computing)Point cloudGroup actionMultiplicationDevice driverVideoconferencingRepresentational state transferFront and back endsComputer animationLecture/Conference
13:26
ImplementationProcess (computing)Module (mathematics)Generic programmingChainGrass (card game)Data structureSocial classPoint cloudGoogolAerodynamicsPoint (geometry)StatisticsArithmetic meanDifferent (Kate Ryan album)MereologyCartesian coordinate systemData managementComputer configurationPoint cloudQuicksortProcess (computing)Line (geometry)Physical lawPresentation of a groupSet (mathematics)Multiplication signPerspective (visual)Information securityCore dumpSoftwareOffice suiteChainBit rateRepetitionConfiguration spaceFitness functionEndliche ModelltheorieBitPhysical systemDivisorInstance (computer science)Open setCodeSatelliteDirection (geometry)Grass (card game)Plug-in (computing)GeometryRepository (publishing)ParsingComputer fileProjective planeModule (mathematics)1 (number)Representational state transferSystem callSoftware repositoryComputer animation
Transcript: English(auto-generated)
00:07
I welcome you to the session. The first talk is by Carmen Tavalica and I'm also co-author here by chance Graz GIS in the Cloud, Actinia Geoprocessing. Carmen is a developer in the company Mundialis in Bonn and heavily involved in the development of this software here.
00:28
She wants to demonstrate to you today. Thank you, Markus. Welcome everyone to my talk, Graz GIS in the Cloud, Actinia Geoprocessing.
00:41
As Markus already said, I'm working for Mundialis. It's located in Bonn, Germany and we focus on processing geo-data. And I worked there since 2015, that's the year when the company was founded, so I'm there from the nearly beginning. And this involved doing a lot of different things, doing web page design and web client building, data processing.
01:09
So I already have a lot of experience in broad topic. Yeah, but then we developed an idea.
01:22
There was a new paradigm occurring with Copernico Sentinel data to bring algorithm to the data. Also, it might be nice in this context to exploit Graz GIS functionality via HTTP REST RP to use it in the internet.
01:42
And then the name occurred, Graz GIS as a service. That was the initial name of Actinia. And this all led to the thinking of how can we put Graz GIS into the Cloud. To understand this, we need to have a little bit know-how about Graz GIS.
02:03
Who of you heard about Graz GIS? Okay, I expected this. And who has worked with Graz GIS? Okay, and I see people developing. Who is developing Graz GIS? Okay, cool.
02:24
So there's one concept we need to understand. One thing is how the Geo data is integrated in Graz GIS and handled. We have the GrazDB which is basically a folder. And inside there we have different locations connected to different EPSG codes.
02:44
And inside these locations we have different map sets. And this would be a project where we're working on. And one map set can have all different types of data. Like raster data, vector data, spatial-temporal data and all other Geo data we can imagine.
03:06
Another thing to understand is that Graz GIS is built up using so-called Graz modules. And there are, I don't know, 300. You can correct me about this, 400. Yeah, and they are all organized to be used with different data.
03:23
If they start with a V, they're used for vector data. If they start with an R, they're used for raster data. And many different tiny ones like FauClip, just clips geometries. But there are also much more complex ones building whole processing chains.
03:45
Okay, how can we bring this into the cloud? We can build a REST API on top of Graz GIS and exploit the locations map sets as resources. We need to enable the usage of the Graz modules, all of them.
04:03
And because we are working in the web, it's good to be somehow managed with user management. So you can sign in and not everyone can access large processing because it costs money to set it up.
04:22
Also, it's important to think on more large scale. I knew that Graz GIS was set up in cluster before. And in the cloud, we have the same problem when multiple users want to access the same data, that there might be concurrency. So we need to implement a locking system.
04:41
And we need to think about the GrazDB management itself. So how does this management work? We have a shared file system where all the data is mounted.
05:04
We have the read-only file system where all the data is located. But to calculate fastly, we mount it in all the Actinia nodes. Actinia nodes is like an installation of what we saw before, the Graz GIS and on top the Actinia REST API.
05:20
And there are more of them. I explain this a little bit later. And the data which we are used for processing is then mounted into the nodes so we can fastly do calculations in the separate nodes. There are different calculation modes. Whereas if we calculate persistent, the data is written back to a user database.
05:43
If we look at this more abstract, we have the Graz database and we have different Actinia nodes. But somehow these nodes need to communicate. This is done with the Redis database. And if a process is started, it is written into the Redis database.
06:03
So the processing node can know the status and update the status. But if somebody requests the status, every node is enabled to look it up in the Redis database. So this is all connected. And to put this into the cloud, we have a lot of work done by cool people to help us with this.
06:28
We don't need to start from scratch. So we use Docker. And Actinia is installed in Docker. And we have different setups already. So it works with Docker Swarm. Everyone who knows Docker Compose, it's really easy just to change the config a little bit
06:45
and say I want to run eight nodes of this. And we also have it in Red Hat with its then called Pods. It's all similar, a little bit different. But it all leads to the usage of having Crust.js in the cloud with different nodes.
07:06
And the user doesn't need to care which node he is talking to. He just requests the system. And then there's a load balancer depending on which software we use, like Docker Compose, Docker Swarm.
07:21
And then it's processing in any of these nodes. Okay, but what is the actual communication that takes place? As I said, it's a REST API. And we can request locations. And if you request this, this is an example response to list available locations in Actinia.
07:43
Or if you want to get more detailed information, what mapsets are in this location, you can also request this. And then it's like going down, the REST API knows that the resources are hierarchically ordered.
08:01
And then in the mapset, permanent, I want to see all raster layers and then I get a list. And there are also some functionalities like render functions so I can see the maps which are already in the Crust database. So then there's a concept of how to work with all the Crust modules
08:21
to not only list locations, we want to do processing. So there are two processing types, ephemeral and persistent. I already said about persistent that calculations can be conducted. And when the results are calculated, they are written back to the Crust database we saw before. There's also the possibility to calculate ephemeral.
08:42
This means that all my results are calculated and then written somewhere where I can download them as a geotiff, for example, or as a shapefile. And on the right you see an example process chain that is one generic endpoint of Actinia where all the Crust modules can be chained and Crusts can be used in the JSON format
09:06
and just sent to Actinia to conduct all the processes that we want to conduct. And there's an example in outputs if I run it ephemeral, I can export it as a geotiff.
09:21
So here's an example. I call this generic endpoint with my process chain. It's this one, it's airslope aspect on a certain map. Then Actinia responds with a JSON. It tells you the status about the processing.
09:42
And one possibility is that I pull the status and just ask how are you, are you still running or what is the status. And when Actinia is finished, the JSON looks different and I can see the results. Another possibility is to tell a web endpoint and then Actinia just notifies when the processing is finished.
10:02
Okay, so now I want to see the results. And then I can request the URLs that Actinia sent me from the processing. There's not only the process chain option where all the Crust modules can be changed. There's also some predefined endpoints which we can use.
10:24
This relates to the idea that I said before to bring the algorithm to the data because Actinia also enables the use of sentinel data and not to download it to their own laptop or to download it all the time.
10:41
But there are different plugins implemented. And examples are to access Amazon RBS or Google Cloud Storage. And they have the advantage that not the whole scene is downloaded but I can choose between different themes. Or even use of ideas platform where I can, when Actinia is installed there, I can mount the data directly and don't need to download it.
11:08
And this plugin can also be used in the process chain when I run importer kind of Actinia module or I can use predefined process chains. One example of a predefined process chain is sentinel 2 process.
11:24
You see there NDVI and then a scene ID. And then we play the same play again. Response with a JSON, poll the status. When it's done, the resources are listed. I get the URLs and then I get the result.
11:41
This is a preview and I didn't upload the large result but it would look like this. Okay. Are there any other features? Yes. There are a lot of them. I won't go into detail into all of them.
12:00
But one nice thing is called RCE, Actinia Command Execution. And the people who work with Crust heavily might feel more comfortable using this because they can just work on the command line, enter a Crust command and then run or configure the Actinia instance they want to calculate with.
12:21
And then they don't execute the commands on the local machine but on the Actinia Cloud instance. And another cool thing is OpenEO. It's an H2020 project and unfortunately at this exact time there's a talk about it so you can't see it but maybe look at it in the videos after.
12:43
And it's a project to develop an API where multiple processing engines are somehow defining a language where the user can conduct processing and doesn't need to know which backend to use but just can easily conduct processing.
13:02
And with OpenEO we developed the OpenEO Cascos driver and because Actinia was already there with a REST API we set this on top so it can be used on the project. Okay. What we want to do? Like every open source project there are always a lot of things to do.
13:27
The first two ones are a little bit technical. It's some kind of process chain management also via REST so that I can create my own process chain and then store it and then execute it all the time and I don't need to save it locally.
13:48
I won't go into details for all of them but other things are like refactoring because the project grew over the time and it wasn't clear at the beginning what would be integrated everywhere so refactoring is always a good thing to make the code more clear.
14:12
Yeah. So great news. Actinia became an OSU community project this year and you can find us on GitHub.
14:23
It's Mundialis, ActiniaCore and there are already some plugins like the satellite plugin I told before or a statistic plugin. And also the API docs are great to get an overview of what Actinia can do and what all the endpoints that can be listed.
14:47
So that was an overview about Actinia. Do you have any questions?
15:09
Yeah, I just want to know because you mentioned there's different options on which software you use to put into the cloud and all that. If I want to install that, my own instance, what do I need?
15:23
Everything is already integrated in the GitHub repo you just saw and the easiest way is to start with Docker and you can find the Docker files for Actinia inside and can try a local setup. And then with Docker Compose there's already latest integrated and when you feel ready you can just switch it to a Docker swarm configuration.
15:46
It's already in this repository as well and then it's like some command line executions and then it's already up and running. Alright, first of all thank you for your presentation, very interesting. Now I
16:02
understand what Actinia is but what I have in my mind is the following. We have OpenEO coming and it seems to be coming quite nicely. Sometime this year we'll have something similar which is the open geo API coming from the OGC that will replace WFS, WCS, WMS, blah blah blah.
16:24
And we also have Actinia. So I see these REST APIs popping up like mushrooms and I would like to understand from your perspective within your company Mandialis, how is this all playing together? I think it all evolves at the same time and can participate from each other. As I said we are as
16:49
well involved in OpenEO and it's not, there are many new things but they can all learn from each other I think. I think it's good to not have only one thing and that's the thing but to have multiple things and they get better. That's my opinion.
17:08
Okay, more questions? Maybe I missed it at the beginning but my question is, is it targeted only to GRASS
17:21
or is it open for other processing tools or applications which could be integrated in these processing chains? It's open. It's designed for GRASS modules but not only. It's already possible to execute Python code which can be uploaded
17:45
and also some command lines like GDAL processing or POI processing are already supported and it's possible to conduct any code.
18:02
More questions? Just to make you understand how this is possible, you know that GRASS add-ons exist and GRASS add-ons use the GRASS parser so you have some dedicated lines in the beginning. And this you can even wrap around whatever other let's say command line tool you can imagine. You just stuff it into it and
18:25
then you could even execute a system call but you have this GRASS parser around which is then delivering all the JSON stuff needed here. So it is pretty easy to do that. It's maybe 10 lines or so and then you wrap around your own code.
18:43
Most command line executions are prohibited because it's a security risk but some can be done already directly.