State of PyWPS
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43436 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Bucharest 2019148 / 295
15
20
28
32
37
38
39
40
41
42
43
44
46
48
52
54
57
69
72
75
83
85
87
88
101
103
105
106
108
111
114
119
122
123
126
129
130
131
132
137
139
140
141
142
143
144
147
148
149
155
157
159
163
166
170
171
179
189
191
192
193
194
195
196
197
202
207
212
213
214
215
216
231
235
251
252
263
287
00:00
State of matterTraffic reportingTable (information)Software developerContent (media)Server (computing)Projective planeService (economics)WebsiteProgramming languageImplementationThermodynamischer ProzessLecture/Conference
00:21
Service (economics)ImplementationThermodynamischer ProzessSpring (hydrology)Operations researchStandard deviationOpen setStandard deviationThermodynamischer ProzessWeb 2.0ImplementationProgramming languageServer (computing)Service (economics)Computer animation
00:44
Thermodynamischer ProzessService (economics)ImplementationSpring (hydrology)Operations researchOpen setStandard deviationService (economics)Web pageExpected valueThermodynamischer ProzessUniform boundedness principleInformationSoftware developerWeb 2.0Computer animation
01:13
Client (computing)Graphical user interfaceProcess (computing)Server (computing)Expected valueLevel (video gaming)CodeProgrammer (hardware)Server (computing)Thermodynamischer ProzessService (economics)Client (computing)Dependent and independent variablesProcess (computing)Multiplication signGraphical user interfaceLine (geometry)Buffer solutionWeb 2.0Vector spaceWeb applicationComputer animation
02:04
Service (economics)Thermodynamischer ProzessStandard deviationDisintegrationSocial classComplex (psychology)Thermodynamischer ProzessInstance (computer science)Computer fileGroup actionMetadataService (economics)Web 2.0Computer animation
02:28
Service (economics)Thermodynamischer ProzessDisintegrationStandard deviationSocial classComplex (psychology)Standard deviationType theoryThermodynamischer ProzessInstance (computer science)Complex (psychology)Revision controlClient (computing)Process (computing)Function (mathematics)CalculationCuboidVector spaceData typeRaster graphicsComputer fileResultantoutputCore dumpComputer animation
03:26
Thermodynamischer ProzessSlide ruleService (economics)InformationFunction (mathematics)TelecommunicationBridging (networking)outputInstance (computer science)Data storage deviceTraffic reportingClient (computing)Process (computing)Thermodynamischer ProzessTelecommunicationEnvelope (mathematics)Functional (mathematics)InternetworkingClient (computing)SoftwareBridging (networking)Web 2.0Standard deviationService (economics)Type theoryComputer animation
04:08
Function (mathematics)outputTelecommunicationBridging (networking)Thermodynamischer ProzessInstance (computer science)Data storage deviceClient (computing)Traffic reportingProcess (computing)Core dumpClient (computing)Computer fileDatabaseArithmetic progressionThermodynamischer ProzessTraffic reportingSource codeInstance (computer science)InternetworkingData storage deviceTelecommunicationServer (computing)CodeCASE <Informatik>Interactive televisionData managementSystem callSet (mathematics)Information securityComputer animation
05:29
Link (knot theory)Grass (card game)Integrated development environmentDisintegrationHome pageRevision controlSoftware developerServer (computing)Client (computing)Java appletScripting languageThermodynamischer ProzessStability theoryImplementationStandard deviationCivil engineeringElectronic mailing listArchitectureCodeLine (geometry)Student's t-testGoogolWebsiteRevision controlProjective planeOcean currentNatural numberComputer animation
06:06
DampingInternetworkingKey (cryptography)Projective planeFamilyRevision controlThermodynamischer ProzessChief information officerINTEGRALGeometryCodeComputer animation
06:46
MehrprozessorsystemInternetworkingParsingDampingLibrary (computing)Electric generatorRevision controlLabour Party (Malta)Computer animation
07:10
MehrprozessorsystemParallel portComputer fileMehrprozessorsystemThermodynamischer ProzessModule (mathematics)Interface (computing)Directed graphComputer animation
07:48
Shape (magazine)Data typeoutputComplex (psychology)Client (computing)Multiplication signOcean currentThermodynamischer ProzessRevision controlServer (computing)Insertion lossGrass (card game)Computer animation
08:21
Data typeComplex (psychology)outputType theoryoutputSocial classNumbering scheme1 (number)Validity (statistics)Data typeLevel (video gaming)Computer fileImplementationClient (computing)Shape (magazine)Regulator geneComputer animation
09:43
Thermodynamischer ProzessServer (computing)Process (computing)Thermodynamischer ProzessStandard deviationSoftware developerInstance (computer science)Directory serviceData storage deviceSystem callResultantGroup actionBitoutputWeightComputer animation
10:09
Thermodynamischer ProzessServer (computing)Process (computing)Integrated development environmentResultantThermodynamischer ProzessFlow separationComputer animation
10:46
Thermodynamischer ProzessModule (mathematics)DatabaseComputer configurationNumberRevision controlCASE <Informatik>Computer fileProcess (computing)Configuration spaceParallel portPolarization (waves)Mechanism designComputer animation
11:22
Computer configurationThermodynamischer ProzessModule (mathematics)DatabaseProcess (computing)Client (computing)Thermodynamischer ProzessHard disk driveDatabaseServer (computing)CalculationMechanism designSystem callNumberQueue (abstract data type)Multiplication signWindowException handlingInterpolationInsertion lossComputer clusterMereologySoftware developerData miningQuicksortNeuroinformatikSampling (statistics)Computer animation
12:53
Function (mathematics)Template (C++)StapeldateiScheduling (computing)Extension (kinesiology)Physical systemSturm's theoremMoment <Mathematik>Thermodynamischer ProzessComplex (psychology)Service (economics)DatabaseData storage deviceHypothesisInterface (computing)MathematicsImplementationSoftware developerPattern languagePoint (geometry)Computer animation
13:49
Thermodynamischer ProzessFunction (mathematics)Complex (psychology)Service (economics)DatabaseData storage deviceHypothesisInterface (computing)Data storage deviceGeometryService (economics)Server (computing)DatabaseProcess (computing)System callResultantStudent's t-testFormal languageProjective planeThermodynamischer ProzessFunction (mathematics)Front and back endsRevision controlClient (computing)FamilyOpen setWeb 2.0Computer fileLevel (video gaming)Presentation of a groupDirectory serviceVector spaceSheaf (mathematics)MereologyOnline helpMultiplication signInstance (computer science)Core dumpGoogolFlow separationPrisoner's dilemmaArchaeological field surveyGroup actionShape (magazine)VideoconferencingInsertion lossShooting methodNatural languageComputer animation
16:27
Broadcast programmingCodeStandard deviationProjective planeCross-correlationGeometryComputer animation
16:47
CodeStandard deviationBroadcast programmingEmailElectronic mailing listGeometryPRINCE2Computer animation
17:11
CloningInstallation artDemo (music)Software developerMereologyGeometryProjective planeLevel (video gaming)Arithmetic meanComputer fileSingle-precision floating-point formatThermodynamischer ProzessConfiguration spaceProcess (computing)Computer animation
18:17
Demo (music)Process (computing)Fluid staticsServer (computing)Template (C++)AreaSoftware testingData structureThermodynamischer ProzessFunction (mathematics)Dependent and independent variablesoutputInheritance (object-oriented programming)Complex (psychology)Process (computing)Thermodynamischer ProzessProjective planeDirectory serviceSingle-precision floating-point formatData structureConfiguration spaceComputer fileoutputElectronic mailing listFunction (mathematics)Social classFunctional (mathematics)Computer animation
18:42
Dependent and independent variablesGeometryShape (magazine)Lattice (order)outputReading (process)AreaFunctional (mathematics)Computer programmingDependent and independent variablesSystem callCASE <Informatik>CodeProgrammer (hardware)Lecture/ConferenceComputer animation
19:07
Dependent and independent variablesGeometryShape (magazine)Lattice (order)outputReading (process)AreaGotcha <Informatik>System callComputer fileProduct (business)File formatUniversal product codePoint (geometry)Video gameShape (magazine)Lecture/ConferenceComputer animationMeeting/InterviewJSONXML
19:32
Usability1 (number)UsabilityCASE <Informatik>Multiplication signPoint (geometry)
20:30
Thermodynamischer ProzessInstance (computer science)Server (computing)Service (economics)CASE <Informatik>Connectivity (graph theory)PurchasingPoint (geometry)Multiplication signRule of inferenceResultantPhysical systemInterface (computing)Electronic mailing listScaling (geometry)DataflowTablet computerOpen setWeb 2.0Shooting methodException handlingEmailNumberTraffic reportingLattice (order)MereologySoftware testingCrash (computing)CodeGotcha <Informatik>Process (computing)Software developerDependent and independent variablesDemo (music)Lecture/Conference
Transcript: English(auto-generated)
00:07
Thank you for coming Let's give you a brief report about the recent and also older development of pywbs project. Oh, there is a table of content. What is pywbs?
00:24
As most of you might know it's the implementation of OGC web processing service standard on the server side using Python programming language. It's rather old It started in 2016 So it, you know, better call it Mature
00:42
and It support, yeah, it uses the older available tools in Python which are, they are, every year there are new tools coming up and pywbs tries to wrap them around and create a WPS or web processing service and develop around them simply and
01:05
Luckily, you can find more informations in a more structured way on the pywbs.org webpage. What is it not? Some people have too high expectations maybe or probably too different expectations.
01:21
It totally shouldn't be complicated for usage from the programmer's perspective. There is no client for web processing services. It's just pure server. I think more. No fancy web application No, even better desktop application, I think. Therefore, yeah, no graphical user interface apparently, so we were heavily inspired by map server and
01:47
There are no processes on the server. People are asking some from time to time does pywbs know how to buffer the responses? Like no, but write the process which will buffer vector data is like two lines of code.
02:06
Nobody did web processing service introduction yet. Very basically, it's one of the OGC XML based files you apparently have get capabilities
02:21
requests and then there are some other metadata requests like describe process and of course the most important one is the execute request which is calling the process instance or starting a process instance on the server and At the end of the day or maybe a next day you as a client should obtain some result of the calculation. There are three
02:45
predefined data types in the version most common version of the standard which is the literal type of data so basically text input or output complex data which is referring to vector data features as well as raster data files and
03:06
bounding box for specifying Yeah, left lower and top upper coordinates pair of coordinates of some region of interest Most used in the practice in practice on daily usage are literal data and complex data
03:22
I don't see much processes using bounding box Yeah This is how it works simply that you have a container with Supporting or offering many processes via the web processing service standard and the client
03:44
Communicates with a container or with the instant with the installation through the three type of request. I was just describing Yeah the essential functionalities of pywps is
04:02
as we said as I said is the communication communication bridge or maybe envelope between The tools and the Internet it fetches the required data from various data sources for for you So to say in a most safe way in a way that it takes care on how big the data are
04:25
And in case they are simply too big it will refuse to download them and to process them as well It creates containers for running processes process instance and tries to make sure that each container don't Interact with each other and that it behaves and on the most
04:42
Secure way as possible because you are at the end of the day executing code on your server So you really want to make sure that you have just executed you can't the code you allowed to be executed Of course, it takes fair under it takes care on the process management itself So communication communication between process between the core and the process itself
05:04
Reporting logging and other Yo, it's storing The data the final process data into some storage which can which can be a file maybe a database where the final
05:20
Datasets are stored saved and of course it Notifies is the client about process status and yeah progress. I Are major It started in 2006 and since then it was developed. We had multiple versions. I
05:46
Believe that I will just skip that so I just wanted to come to the recent version Because Sorry
06:01
Here we go Officially of pywps4. This is the current version Here we go And so today we are talking about pywps4. There was a big restructure realization of the code and We graduated last year as OS geo project. We were probably the longest project in the incubation process
06:26
It took us nine years. I always said it proudly and loudly it took by nine years to graduate to as Joe but now we are Well official OS geo project and happy member of the OS geo family and
06:43
What are the key features of pywps4? Because pywps4 was written from scratch from the previous version more or less more more than less Because of they were new technologies flying around on the internet
07:00
Which could and should be used as well, for example the L XML library which makes the parsing and Generating of XML Text files much more easier and faster as well We are using On heavily depending on flask as well
07:20
There is this WSGSI interface which had to be somehow yeah addressed A new multi-processing modules for parallel process calls Yeah, and of course Docker it's one of the technologies you totally probably need to address as well because of the containers and it's
07:45
It seems to be very promising technology In pywps4 we finally are Validating some data in the previous versions. We didn't care much. What did the client sense? Send is sending to the server as long as the data could be processed
08:05
by the underlying tool like GL for example or Grass GIS as long as it could be imported. We assume that the data are okay with the current version user though Yeah user of pywps can set for each process
08:25
Level of validation of the input data. So basic for each data type The basic is of course no validation and then there is also the the most strict one is very strict validation Which uses the XML scheme, I I guess that most of the
08:45
Implementations out there or most of the usages will just stick to strict validation which just attempts to Compare the mime types and probably and Make sure that it fits somehow that if do you if the client claims that there is a JSON file going in then?
09:05
It's probably a JSON file as well But if you need to be strict to and because of any reason you can go with XML scheme validation as well this of course applies just for GML inputs type of inputs if
09:21
You for some reason needs to send sri shapefile or gorgeous nor gml the validators need to be adjusted accordingly and Pywps is currently having a validator classes for these type of inputs the other ones would have to me Would have to be at
09:49
How our process containerized there is a lots of happening in the recent development the standard weight is that
10:01
The process the Pywps instance on every execute call creates a separated Directory and within this directory all the data or the input data are downloaded as well as or stored as well as Result of the execution is stored down as well with Pywps for we try to
10:22
We aim for safer approach like using docker container so that every process runs in separated docker environment and At the end of the day So that there is no there is no Danger that
10:41
The processes will simply rewrite each other's data stuff like that There is also a new mechanism of How the polarization is handled in the previous version the only way how you could
11:03
Steer the number of processes which are allowed to run in parallel Was just by a number number of processes which can simply run in parallel in the configuration file in this case There was a big issue if you want if you need it for some reason to have
11:22
If you had a probe if you had the process which would interpolate digital elevation model Which would be very much a computer. Sorry processor demanding as well as hard drive demanding and If there would be four calls in parallel and the number of processes allowed would be for example three
11:42
Then the last one would be just thrown away so to say it wouldn't be allowed allowed to run on the server With pywps4 We firstly about onto the usage of OS fork, which we didn't which wasn't working on on windows of course, but there is a
12:06
Mechanism which will store the requests and put them in the queue so and all the time some process well and the calculation it will just look in the Database whether there is another whereas there is whether there is another request for father
12:23
process execution and then it would just take The last call from the stack and continue with the process execution so in this way we can store 50 requests for example in the queue and So that the client
12:42
shouldn't get The Exception that there are too many processes on the server being executed much too often yeah, now the development what was happening recently is that we
13:00
Completely have returned back to template based XML Rendering Which is much faster and basically also very much easier to be implemented that would be probably the biggest change and This is where we are right now
13:22
We have what we would be like What would we like to do? implementation of wps2 0 it is around for a couple of years already Let's keep I don't know the details, but we are giving this talked on many phosphor g-cent every year last
13:41
In last couple of years. We are saying yeah, we would like to implement wps 2.0 I Personally doubt that we are going to do that We are probably skipping we are probably going to skip this and we are going to for the open API Directly, so let's call it wps3 According to the wfs 3 pattern, but the features which will be needed are
14:07
the features which Wps2 introduces will be needed anyway. It is like pausing of the processes Maybe deleting of the running processes stuff like that has to be implemented and we have to went best way how to do how to
14:20
Achieve this with Python docker containers flying all around. I was mentioning this several times we still I'm still personally not satisfied with the way how it this is done and I would like to incorporate docker instances more within the pywps core
14:42
one of my dreams which was in the version of the web processing service sorry in pywps 1 0 was that the output of the process Shouldn't be only a data file like for example geotiff for shapefile or gML
15:00
But it could be a service directly so that the client could make a request okay I want please process my My vector data and result should shouldn't be gML But the result should be wfs endpoint generating on the fly this was in the first version of pywps, and we didn't really implement it so far but
15:24
One of the ideas was that this would be implemented with help of Either map server or geo server preferably both of course as a back-end engines I? Was already mentioning geo API or open API sorry for
15:42
Yeah, my mistake There was a presentation in the previous section about it and pywps totally needs to be part of the family Our students of Google summer of cold Projects in recent years make some effort for the output storages
16:02
I was mentioning that usually the output goes to Directories or but but students were working heavily on having the have it a possibility to store the outputs to either Amazon S3 storage or maybe to post GIS database
16:21
Yeah support for the languages of course and other How is the project? organized We are OG OS geo project that means we have a project steering committee some of the members of the committee are here in this room like myself
16:43
Don't correlate is sitting behind Most of you. Thank you for coming and Luis Chairing the session in the other room If you want to join us if you want to join our community either go to getter
17:03
Which is probably the preferably the preferably place to have chat and to come in communicate as well as you can join us on the mailing list on OS geo and We are trying to attend sprints and conferences as for sport G What I personally say we are failing in is fundraising
17:26
Yeah We are incubated OS geo project that guarantees certain level of Sustainability and that we have a healthy and growing community. That means we
17:40
But it doesn't mean that we can give up and say okay. This is the given statues or granted we have to work on this heavily and Find new resources for the further development as well Wow, I have still three minutes. So very fastly how to get it up
18:02
install preferably with install This is how the how the Installation looks like and the most important part is By WPS configuration file along with processes directory where all the processes can be started as single Python files
18:27
This is the project structure. I'm sorry process structure a process called aria Is a class derived from process and it has list of inputs as well as list of outputs and
18:41
at the end there somewhere has to be a executing function Which obtains requests and response? and This is the code you should be able to write in a case You are a Python programmer and this should be relatively easy for you to understand. So what I see here. Oh
19:03
This is nasty. I would never do that. But if you if you totally need to do this if you need to transform Some file to the other format and use ogr2 ogr. This is possible, too So let's say this demonstrate the possibilities. It has not necessarily I would call this a production code
19:24
Very briefly about the philosophy still one minute Oh Louis loves idea of bikes. I love bike tools. So This tries to demonstrate that Pywps is like a bike apparently we have bike in the logo bike can be
19:43
By can have different shapes, but still can bring you from point a to 4 point B There can be large there can be small, of course They can be very cheap once but expensive and fancy ones There can be very simple and very complex. I have rather preferred a simple one in this case
20:06
If I would speak about pywps So this all describes our philosophy it should be versatile Pywps can be whatever you want it to be and it should be very much ease of use and easy to or self self-repair
20:24
You just need to know how to pedal Question time. Thank you. In case you have no question. I have some but please
20:42
So my questions well technical again, but how come you chose to use? standalone docker docker containers as opposed to Services because that way you are practically reduced using the resources on one system while if you would have services like for I know the little using docker swarm or Kubernetes you would not be
21:04
Restricted one system, but to a cluster which can be grown or grown or Shrink and automatically and also the number of services that are on automatically So it just seemed okay, perhaps it's not that easy, but it's much more easy to scale everything like that
21:24
Thank you for your comment, I'm not personally involved in this in this part of code So we will be welcome to join us and the mailing place latest I would like to see some demos as well from what I know one of the there. There are two places where you can
21:41
Put docker in or inside of pywps either you can containerize You can containerize the whole Instance or what we are trying to do is that each process instance starts its own docker container on demand Deploys it somewhere and this will be probably or point if I understand correctly and then we
22:04
Reuse the web processing service interface to call inside a process one more time Okay, I Was just saying that if you use it as a service it would be more scalable and
22:23
Possibly less resource resource hungry because you don't have to have already prepared something Basically, if you need something you'll be spawned if you don't need it. You will be closed down. So that's what I was Kind of okay. That's More technical. Thank you
22:42
Thanks for the comments as well and yeah, so imagine you run a process and you spawn a docker container or something If you run a process and you spawn a docker container and then your server crashes what happens to your processes
23:01
Can you actually get the results? When the question was if I understand correctly what if I run it in the container and the server crashes? Like the instance of the server. Yes I mean if say you send a request which takes two days to process and After two hours the web server crashes the BPS server crashes, but your process is still running
23:25
Can you get back to the process somehow? Any handle the server crashing and coming back up again, you know, no once it crashed the you should immediately obtain a
23:42
Execution exception report and Pi WPS tries as hard as possible to clean everything behind itself so that there are no data Lying around and no zombie processes running around same applies for the docker as far as I know these Docker was was one of the possibilities we were looking for by the way also for the process posing as well as process killing
24:08
So So one more time to repeat my answer if some process crashes We try to kill it and clean afterwards everything as hard as possible If it answers your question
24:25
Or if this isn't okay. Yeah, Jana has one Do you have a users list as well as a developers mailing list if I just want to use Pike WPS rather than developing it
24:45
Do you have a mailing list I can use? Sorry. Yeah, that was a easy one. No But but but yes But yes, they are they are we are we are not we there is not like
25:00
400 people at a mailing list You don't have to be afraid of two technical questions lying all around You just can join and pretend you are normal user and you will be kind To as well as if you join Gitter Then you might get some response from some of the community members in much nicer way Thank you
25:21
More questions still If it isn't the case, thank you for your attention and see you here or in other rooms in five minutes