WPS services with QGIS Processing
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43458 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Server (computing)ResultantThermodynamischer ProzessWordPresentation of a groupWeb 2.0Workstation <Musikinstrument>Lecture/ConferenceMeeting/Interview
00:42
Web serviceServer (computing)Process (computing)Web serviceServer (computing)AlgorithmThermodynamischer ProzessProjective planeWeb 2.0Function (mathematics)Process (computing)Set (mathematics)Source codeMereologyCASE <Informatik>NumberPlastikkarteProduct (business)Computer animation
02:39
Software developerWeb serviceServer (computing)FrequencyTraffic reportingParameter (computer programming)Context awarenessSoftware developerTexture mappingFront and back endsProjective planeWeb 2.0Server (computing)Context awarenessComplex (psychology)Mathematical analysisWeb browserEndliche ModelltheorieIntegrated development environmentTraffic reportingClient (computing)Point (geometry)Perspective (visual)MappingCASE <Informatik>Computer animation
03:31
Integrated development environmentFrequencyTraffic reportingParameter (computer programming)Context awarenessDecision theoryParameter (computer programming)Point (geometry)outputEndliche ModelltheorieCASE <Informatik>FrequencyTraffic reportingGroup actionMultiplication signAlgebraComputer animation
04:45
Axiom of choiceDecision theoryContext awarenessoutputEndliche ModelltheorieAlgorithmWeb serviceSmoothingData analysisProcess (computing)CASE <Informatik>Software developerDecision theoryProcess (computing)Context awarenessCartesian coordinate systemAreaObservational studyServer (computing)Traffic reportingFluid staticsEndliche ModelltheorieFunction (mathematics)AlgorithmResultantComputer animation
06:02
SmoothingEndliche ModelltheorieData analysisoutputAlgorithmWeb serviceProcess (computing)Context awarenessDependent and independent variablesThermodynamischer ProzessInterface (computing)Standard deviationRule of inferenceServer (computing)Parameter (computer programming)Source codeMultiplicationSoftware developerProcess (computing)Computer simulationServer (computing)AlgorithmSource codeProjective planeThermodynamischer ProzessWeb serviceWeb 2.0outputFunction (mathematics)Functional (mathematics)Descriptive statisticsCode refactoringPlug-in (computing)Level (video gaming)Point (geometry)CodeFocus (optics)Cellular automatonBootingSound effectObservational studyOrder (biology)Pattern languageComputer animationProgram flowchart
10:00
Server (computing)Thermodynamischer ProzessMultiplicationProcess (computing)Root2 (number)Channel capacityLattice (order)KnotBit rateServer (computing)Thermodynamischer ProzessResultantImplementationCommunications protocolMultiplicationSoftware frameworkComputer animation
10:57
ScalabilityMappingExtension (kinesiology)Regulärer Ausdruck <Textverarbeitung>Thermodynamischer ProzessDependent and independent variablesSynchronizationServer (computing)Process (computing)Order (biology)Multiplication signRevision controlData structureMereologyMathematicsWeb serviceUsabilityPosition operatorCellular automatonNetwork topologyMatching (graph theory)BitParticle systemFreewareFigurate numberComputer animation
13:41
Front and back endsScalabilityMappingExtension (kinesiology)Regulärer Ausdruck <Textverarbeitung>Server (computing)Process (computing)Streaming mediaExtension (kinesiology)Multiplication signResultantShape (magazine)BitData storage deviceDatabaseCache (computing)Selectivity (electronic)ExpressionoutputSheaf (mathematics)Staff (military)Thermodynamischer ProzessTexture mappingBit rateComputer animation
15:20
Process (computing)Parameter (computer programming)outputMappingContext awarenessFunction (mathematics)Personal digital assistantString (computer science)Server (computing)Thermodynamischer ProzessCASE <Informatik>Parameter (computer programming)BitTerm (mathematics)Multiplication signComplex (psychology)Context awarenessPosition operatorOrder (biology)outputInformationTexture mappingComputer animation
15:56
Server (computing)Function (mathematics)outputString (computer science)Context awarenessMappingParameter (computer programming)Process (computing)Personal digital assistantServer (computing)Texture mappingFunction (mathematics)Projective planeResultantInformationLatent heatOffice suiteInheritance (object-oriented programming)Computer animation
16:42
Internet service providerServer (computing)AlgorithmSource codeMappingProjective planeResultantWordPosition operatorAlgorithmDependent and independent variablesSource codeCausalityComputer animation
17:01
Server (computing)CodeThermodynamischer ProzessSource codeResultantAlgorithmServer (computing)Process (computing)Projective planeOrder (biology)Lecture/ConferenceComputer animation
17:53
Process (computing)Error messageJapanese honorificsSoftware testingContext-sensitive languageThermodynamischer ProzessInformationProcess (computing)State of matterArithmetic progressionSource codeXML
18:23
Process (computing)State of matterThermodynamischer ProzessInformationError messageLevel (video gaming)Process (computing)Texture mappingDivision (mathematics)Source code
18:41
AlgorithmoutputParameter (computer programming)Client (computing)Server (computing)Visualization (computer graphics)FacebookServer (computing)Extension (kinesiology)
19:05
Server (computing)Multiplication signPoint (geometry)View (database)Projective planeCartesian coordinate systemOpen sourceServer (computing)Descriptive statisticsProcess (computing)State of matterResultantComputer fileMereologyDemo (music)Source codeSet (mathematics)Order (biology)Right angleNumberLecture/ConferenceMeeting/InterviewXML
20:42
Server (computing)Demo (music)Computer fileCellular automatonServer (computing)Medical imagingQueue (abstract data type)Client (computing)XMLLecture/ConferenceComputer animation
21:23
Server (computing)Proxy serverSet (mathematics)CodeSystem callCellular automatonClient (computing)Multiplication signLecture/Conference
Transcript: English(auto-generated)
00:07
Next presenter in this session of this web processing session, David Martin from France and if I hear co-corrected and you may see it already, this will be about QGIS and WPS.
00:25
Okay. Thank you for coming to this session. Okay. I'm David Martin from Trolleys, Trolleys which is a small company in France. We provide GIS services and we come to a delta. Just a few words on this presentation. As you may know or not,
00:44
you could now, you can leverage web services with QGIS server, thanks to QGIS server. The nice fact with that is that it is as simple as just taking the QGIS project that you have on the desktop,
01:03
that you have elaborated with QGIS desktop, and put it on the server and then if QGIS server is running there, you have your publication, your web service which is published. What we're going to show here is through a typical use case,
01:24
is that we have implemented exactly the same paradigm, but for web processing service and from QGIS processing algorithms. Everybody know what is QGIS processing algorithm?
01:41
No? Yes? More or less? Okay. Algorithm or processing algorithm in QGIS or a set of an API, which enable you to run any algorithm in QGIS, that operate on your data, on your layers and so on, and that produce output which can be layers or any kind of
02:05
geospatial data that you have been processing. So what we are doing here is we are doing the same thing for web process service as you elaborate, you can test and create your QGIS algorithm on the desktop,
02:22
then you take your code, your processing code, QGIS processing code, you put it on the server and you have your web service, your web processing service published. Okay. So let's go to some introduction.
02:40
Okay. So Twiliz, which is a French company, we are a developer of Lismap. I don't know if you know that. It's a web client which is a web map client, which is based on QGIS server. We also provide GIS hosting services based on Lismap and QGIS as backend.
03:03
And we are also contributor to QGIS and QGIS server. Okay. So some context of this project. This is a project which has been supported financially by NFE, NFE which is a French company that mainly do expertise by providing models,
03:27
complex models for prospective analysis of environmental economic impact in this case of deforestation. That is building a tool from which they create reports
03:40
that are used by stakeholders to measure the impact of decision taken by decision makers usually. But they face two problems here. First, data are changing on relatively small time periods. That is the time you produce the report, the data which has been used for this report
04:03
is already out of date. Then these models are very complicated. And believe me, when I say complicated, these models are using multiple parameters for their input that should simulate the decisions,
04:24
the parameters simulate the decisions that has been taken. And so the models are driven by this parameter. This parameter simulates the action of the decision makers. As you know, decision makers can vary rapidly too. So in this case, they need the tools where at some point
04:44
you can run your model and feed them in a very efficient way to get as much as rapidly as possible the results and the dashboard which gives you
05:02
the impact of the decision that could be taken or has been taken. OK. Obviously, they work with QGIS and QGIS processing. That is, they elaborate their models with error, for example.
05:24
And they use QGIS to run them and to produce the output for the reports. So they need, in fact, what they want to do is not providing a kind of static report, but let the stakeholder feed the models as, in fact,
05:46
depending on the context of the political and environmental context. So in this case, what we want, we want to expose, it seems too natural to expose this algorithm as WPS services.
06:00
OK, so in this case, they need a smooth workflow from development of computational model on QGIS to their publication process on the server. OK, web processor, I won't go here because I suppose that everybody knows about WPS. OK, so this has been, OK, this is all the way through.
06:25
I won't go on this again because it had just been described just before. But WPS service consists of three main requests, which are get capabilities, describe process, and execute.
06:40
So what's, OK, how QGIS processing work? You have your data source, which are presented by QGIS project, which describe which layers, which data source you're using, and so on. You have your processing algorithm, which is a code, a Python code, which use the processing API
07:04
from QGIS that take the inputs that describe the kind of input used and the kind of output that the algorithm provide. This is, in fact, we feel that is
07:22
quite very similar in the functionality as the WPS. We are very close. You need to have a process where you need to give the description of the input and the description of the output. Then once you run from QGIS desktop, you run the algorithm.
07:41
And you produce output, which go to data sinks, which can be databases, layers, and so on. OK. So what we want to do is this. That is, we have the data source QGIS project on the server that describes all the data sources.
08:01
We have the processing algorithms. Exactly the same, the very same that the one we are using with QGIS desktop. We have just taken the same Python code. We just put it on the server. And we produce data sinks, which can be databases, inputs, or layers, or shapefile,
08:22
or anything we want, and also QGIS projects that describe all that. But here, we operate the processing algorithms not from QGIS desktop, but from the WPS service. So what we need to do, mainly,
08:41
is to marshal the inputs of the WPS to processing inputs, and then doing the reverse, marshaling processing outputs to WPS outputs. How we have done this?
09:03
There is, OK, we could have, for example, we could have used QGIS server with QGIS3, with the refactoring of QGIS3. We, QGIS server, is able now to provide multiple services. You can add services as plug-in, as first-class plug-in.
09:24
That is, actually, we have WVMS, WVFS, WVCS, but we could also, I think, to add a new WPS service at the server level. It's not easy. I will go to that later. So this has been done at some point
09:43
with an old, very old plug-in, which is WVPS for server, which has been used for implementing the WVPS service at the server level. But the things we have to consider here is that WVPS is asynchronous by design.
10:03
That is, you have to run multiple servers, multiple processes at the same time, and in an asynchronous way, you have to give the results and the status of your run. So, but QGIS server, by construct, is synchronous.
10:21
So it's a problem. Then you need to have a multiprocessing framework to run your process. Not shredding, but multiprocessing. And that is not possible to make it in a QGIS server.
10:45
So we need to redesign an implementation of the WVPS protocols and run as backend, processing as a backend. So we do not want to reinvent the wheel. So what we have done, we have forked,
11:01
we have used a fork of PWPS in this earlier version 4. And I will explain why we have done that. Because we had requirements that were not easy
11:21
to match with the actual structure of PWPS. So what we were interested in, it is mostly the parts which deal with the WVPS protocol, which was nice and nicely done. So it was cool to reuse that part.
11:43
Mainly the change from PWPS that we have done is it's mostly QGIS-centric. This we do not support as PWPS support any kind of process that you want. It's not an API. It's really a server which serves processing
12:03
as a WVPS service. So it's QGIS-centric. All processes run QGIS processing. We have changed the way it runs. PWPS is based on the WZR. Here we needed to have an asynchronous request only.
12:23
That is, even there is two ways in PWPS to ask for an execution of a process. The first is you can ask in a synchronous way. You ask the server and you wait for the response.
12:42
The other way is you ask for the server and you poll the server regularly to get the status of the running process. That is, if you make a synchronous request in order to wait for, you don't want your server
13:02
to be blocked by the process at run. So you need, at the server level, a way to have an asynchronous processing of the request. We wanted to support Python 3. We don't want to support Python 2 because QGIS has been rewritten with Python 3 support.
13:25
So we have dropped, totally, the Python 2 support. When we forked, at the time we forked, PWPS was still supporting Python 2. So this is something we do not want here. And we wanted to use, as most as possible,
13:41
the new AsyncQ API of Python 3. We wanted to have something which is just a bit more scalable, that is actually PWPS store intermediate results and status of the process in a database. So here it was not really adequate with our infrastructure.
14:06
So we wanted to support Redis. Instead of using the database, we use a Redis cache to store the intermediate status of the process. We wanted to support stream and chunk request too,
14:21
which is not easy to do with WSGI. And we have some extension in mind, which was map. We go to that, I will go to that. Timeout, that is you want, if you have a process that take longer, that some time out you just want to kill it.
14:41
And expire, that is when you produce data, the data will stay on the server, as it could be a shape file, it could be anything. But you may require that these results are obsoleted after a certain amount of time,
15:01
so it should be deleted from the server. And we wanted some more goodies, which is more QGIS related, for example, feature selections with QGIS expression and things like that, and also status API. Okay, so the first thing to do
15:21
was marshaling the WPPS input-output to and from processing. That is, processing is much, a bit is rich, is more rich, in terms of kind of parameters that you can handle as input.
15:40
So in this case we need some time to abuse of complex input-output for indeed very special case. We need to provide a context, because in QGIS when you run processing, you run, you have a QGIS project, which hold all the information about the data that you're working on.
16:01
So we need a map parameter, which does not exist, I think, in the WPPS specifications, but which is used by QGIS, actually it's used also in WVMS and WVFS requests. So, and we wanted to return output layers
16:24
as WVMS and WVCS references. That is, in fact, when you have a project, you operate on that project, on the data of that project, then you create another project with all the results. So, in fact, you can do exactly the same thing,
16:41
that you can use QGIS server on that newly created project to do the request, WVMS request, to see the results in WVFS request to get the results of the processing. So, this is a schematic way of what happened. You have the algorithm provider,
17:03
which is the Python code of the QGIS processing. You, the WVSServer, you do the WPPS request, you have a source project, which is a QGIS project, then you generate the results project, which hold all the data which has been produced
17:23
by running your processing algorithms. Then, what you can do, and what you're doing here, is that we use QGIS server to publish, in fact, the results that have been generated. So, we stay in QGIS in all the process.
17:42
We can also get this QGIS project and get it back into QGIS desktop, which could be very convenient. Also, we have a new place. We need to look at the state of the process.
18:01
So, we have developed a statute, HTTP REST API, that which is used to know the state of the process. If it is finished, here it's not showed, but you have progress bar that indicate the progress of the running process,
18:23
because in QGIS processing, you can send information about the state of the process. And you can know if there is an error that has been raised at the QGIS processing level. We use Lismap, so, as a Lismap developer,
18:45
we have developed an extension of Lismap that can do a WPS request to the server and visualize them through a QGIS server request, like WMS or WFS requests.
19:03
Okay, so, I'm done. So, thanks for my attention. And thank you for being perfect on time. I was happy to see one of the bike modifications to PyWPS you did. Any questions?
19:27
Please. So, for the expired part, I saw that you also provided the expired part for that data, right?
19:42
In the server. Expired part. So, the data should expire after a certain time. Yes. Okay. Since you used Redis, Redis already has that, right? Redis has that for the data, it also. But in Redis, what we store is just description of the state of the process.
20:01
Okay, so. Not the file which has been generated. It's only the, okay. We have an expired time in Redis, but we have also, we use this at thank you timers to get. Okay, so the initial data is not there, right. Which is generated when the timer occurs. Okay, got it. And delete the results that are stored on the server.
20:23
Okay, thanks. Thank you. More? Is there some demo application where we could try it out? Demo application, public one? There is, all the project is open source.
20:43
Of course, but is there a demo in, sorry. No, there is no demo actually. But you can run, we have Docker. We have some Docker files that is used to build ready to use a server.
21:01
And also we have some, so you can find it, there's a Docker file here and you can find a ready Docker image on Docker Hub. But we have no demo yet. Thank you. More questions?
21:23
Did you make some work on the QGIS WPS client? Could this WPS client? Yeah, did you contribute or maybe do you want to contribute to the old WPSQGIS client?
21:43
Could, but I did not work on it. Just asking. But the idea is to, because it's the same code, so we share QGIS desktop and QGIS in WPSQGIS server
22:00
share exactly the same set code which also we can use. You can use it to delegate work from QGIS desktop to server, for example. Something like that, anyway. Well, last call for questions. Everybody is satisfied. Thank you very much, one more time. Thank you. And yeah, see you here. Thank you. Thank you.