We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

State of PyWPS

00:00

Formal Metadata

Title
State of PyWPS
Title of Series
Number of Parts
295
Author
Contributors
License
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
PyWPS is an open source, light-weight, Python based, implementation of the OGC Web Processing Service (WPS) standard. It provides users with a relatively seamless environment where to code geo-spatial functions and models that are readily exposed to the Internet through the WWW. Initially started in 2006, the current 4.x version takes advantage of the state-of-the-art Python infrastructure in order to provide new and useful features. The current version implements the WPS 1.0 standard entirely. The WPS 2.0.0 and it's features is currently on-hold status. PyWPS offers a straightforward WPS development framework with the increasingly popular Python language. Python offers access to a vast array of code libraries that can be easily used in the processes, in particular those for geo-spatial data manipulation, e.g. GRASS, GDAL/OGR, Fiona, Shapely, etc., but also to statistics packages (e.g. rpy2 for R statistics) and data analysis tools (e.g. pandas). PyWPS offers storage mechanisms for process inputs and outputs and spawns processes to the background for asynchronous execution requests. Talk will present features of the last 12 months of development and outline the roadmap for next versions of PyWPS.
Keywords
129
131
137
139
Thumbnail
28:17
State of matterTraffic reportingTable (information)Software developerContent (media)Server (computing)Projective planeService (economics)WebsiteProgramming languageImplementationThermodynamischer ProzessLecture/Conference
Service (economics)ImplementationThermodynamischer ProzessSpring (hydrology)Operations researchStandard deviationOpen setStandard deviationThermodynamischer ProzessWeb 2.0ImplementationProgramming languageServer (computing)Service (economics)Computer animation
Thermodynamischer ProzessService (economics)ImplementationSpring (hydrology)Operations researchOpen setStandard deviationService (economics)Web pageExpected valueThermodynamischer ProzessUniform boundedness principleInformationSoftware developerWeb 2.0Computer animation
Client (computing)Graphical user interfaceProcess (computing)Server (computing)Expected valueLevel (video gaming)CodeProgrammer (hardware)Server (computing)Thermodynamischer ProzessService (economics)Client (computing)Dependent and independent variablesProcess (computing)Multiplication signGraphical user interfaceLine (geometry)Buffer solutionWeb 2.0Vector spaceWeb applicationComputer animation
Service (economics)Thermodynamischer ProzessStandard deviationDisintegrationSocial classComplex (psychology)Thermodynamischer ProzessInstance (computer science)Computer fileGroup actionMetadataService (economics)Web 2.0Computer animation
Service (economics)Thermodynamischer ProzessDisintegrationStandard deviationSocial classComplex (psychology)Standard deviationType theoryThermodynamischer ProzessInstance (computer science)Complex (psychology)Revision controlClient (computing)Process (computing)Function (mathematics)CalculationCuboidVector spaceData typeRaster graphicsComputer fileResultantoutputCore dumpComputer animation
Thermodynamischer ProzessSlide ruleService (economics)InformationFunction (mathematics)TelecommunicationBridging (networking)outputInstance (computer science)Data storage deviceTraffic reportingClient (computing)Process (computing)Thermodynamischer ProzessTelecommunicationEnvelope (mathematics)Functional (mathematics)InternetworkingClient (computing)SoftwareBridging (networking)Web 2.0Standard deviationService (economics)Type theoryComputer animation
Function (mathematics)outputTelecommunicationBridging (networking)Thermodynamischer ProzessInstance (computer science)Data storage deviceClient (computing)Traffic reportingProcess (computing)Core dumpClient (computing)Computer fileDatabaseArithmetic progressionThermodynamischer ProzessTraffic reportingSource codeInstance (computer science)InternetworkingData storage deviceTelecommunicationServer (computing)CodeCASE <Informatik>Interactive televisionData managementSystem callSet (mathematics)Information securityComputer animation
Link (knot theory)Grass (card game)Integrated development environmentDisintegrationHome pageRevision controlSoftware developerServer (computing)Client (computing)Java appletScripting languageThermodynamischer ProzessStability theoryImplementationStandard deviationCivil engineeringElectronic mailing listArchitectureCodeLine (geometry)Student's t-testGoogolWebsiteRevision controlProjective planeOcean currentNatural numberComputer animation
DampingInternetworkingKey (cryptography)Projective planeFamilyRevision controlThermodynamischer ProzessChief information officerINTEGRALGeometryCodeComputer animation
MehrprozessorsystemInternetworkingParsingDampingLibrary (computing)Electric generatorRevision controlLabour Party (Malta)Computer animation
MehrprozessorsystemParallel portComputer fileMehrprozessorsystemThermodynamischer ProzessModule (mathematics)Interface (computing)Directed graphComputer animation
Shape (magazine)Data typeoutputComplex (psychology)Client (computing)Multiplication signOcean currentThermodynamischer ProzessRevision controlServer (computing)Insertion lossGrass (card game)Computer animation
Data typeComplex (psychology)outputType theoryoutputSocial classNumbering scheme1 (number)Validity (statistics)Data typeLevel (video gaming)Computer fileImplementationClient (computing)Shape (magazine)Regulator geneComputer animation
Thermodynamischer ProzessServer (computing)Process (computing)Thermodynamischer ProzessStandard deviationSoftware developerInstance (computer science)Directory serviceData storage deviceSystem callResultantGroup actionBitoutputWeightComputer animation
Thermodynamischer ProzessServer (computing)Process (computing)Integrated development environmentResultantThermodynamischer ProzessFlow separationComputer animation
Thermodynamischer ProzessModule (mathematics)DatabaseComputer configurationNumberRevision controlCASE <Informatik>Computer fileProcess (computing)Configuration spaceParallel portPolarization (waves)Mechanism designComputer animation
Computer configurationThermodynamischer ProzessModule (mathematics)DatabaseProcess (computing)Client (computing)Thermodynamischer ProzessHard disk driveDatabaseServer (computing)CalculationMechanism designSystem callNumberQueue (abstract data type)Multiplication signWindowException handlingInterpolationInsertion lossComputer clusterMereologySoftware developerData miningQuicksortNeuroinformatikSampling (statistics)Computer animation
Function (mathematics)Template (C++)StapeldateiScheduling (computing)Extension (kinesiology)Physical systemSturm's theoremMoment <Mathematik>Thermodynamischer ProzessComplex (psychology)Service (economics)DatabaseData storage deviceHypothesisInterface (computing)MathematicsImplementationSoftware developerPattern languagePoint (geometry)Computer animation
Thermodynamischer ProzessFunction (mathematics)Complex (psychology)Service (economics)DatabaseData storage deviceHypothesisInterface (computing)Data storage deviceGeometryService (economics)Server (computing)DatabaseProcess (computing)System callResultantStudent's t-testFormal languageProjective planeThermodynamischer ProzessFunction (mathematics)Front and back endsRevision controlClient (computing)FamilyOpen setWeb 2.0Computer fileLevel (video gaming)Presentation of a groupDirectory serviceVector spaceSheaf (mathematics)MereologyOnline helpMultiplication signInstance (computer science)Core dumpGoogolFlow separationPrisoner's dilemmaArchaeological field surveyGroup actionShape (magazine)VideoconferencingInsertion lossShooting methodNatural languageComputer animation
Broadcast programmingCodeStandard deviationProjective planeCross-correlationGeometryComputer animation
CodeStandard deviationBroadcast programmingEmailElectronic mailing listGeometryPRINCE2Computer animation
CloningInstallation artDemo (music)Software developerMereologyGeometryProjective planeLevel (video gaming)Arithmetic meanComputer fileSingle-precision floating-point formatThermodynamischer ProzessConfiguration spaceProcess (computing)Computer animation
Demo (music)Process (computing)Fluid staticsServer (computing)Template (C++)AreaSoftware testingData structureThermodynamischer ProzessFunction (mathematics)Dependent and independent variablesoutputInheritance (object-oriented programming)Complex (psychology)Process (computing)Thermodynamischer ProzessProjective planeDirectory serviceSingle-precision floating-point formatData structureConfiguration spaceComputer fileoutputElectronic mailing listFunction (mathematics)Social classFunctional (mathematics)Computer animation
Dependent and independent variablesGeometryShape (magazine)Lattice (order)outputReading (process)AreaFunctional (mathematics)Computer programmingDependent and independent variablesSystem callCASE <Informatik>CodeProgrammer (hardware)Lecture/ConferenceComputer animation
Dependent and independent variablesGeometryShape (magazine)Lattice (order)outputReading (process)AreaGotcha <Informatik>System callComputer fileProduct (business)File formatUniversal product codePoint (geometry)Video gameShape (magazine)Lecture/ConferenceComputer animationMeeting/InterviewJSONXML
Usability1 (number)UsabilityCASE <Informatik>Multiplication signPoint (geometry)
Thermodynamischer ProzessInstance (computer science)Server (computing)Service (economics)CASE <Informatik>Connectivity (graph theory)PurchasingPoint (geometry)Multiplication signRule of inferenceResultantPhysical systemInterface (computing)Electronic mailing listScaling (geometry)DataflowTablet computerOpen setWeb 2.0Shooting methodException handlingEmailNumberTraffic reportingLattice (order)MereologySoftware testingCrash (computing)CodeGotcha <Informatik>Process (computing)Software developerDependent and independent variablesDemo (music)Lecture/Conference
Transcript: English(auto-generated)
Thank you for coming Let's give you a brief report about the recent and also older development of pywbs project. Oh, there is a table of content. What is pywbs?
As most of you might know it's the implementation of OGC web processing service standard on the server side using Python programming language. It's rather old It started in 2016 So it, you know, better call it Mature
and It support, yeah, it uses the older available tools in Python which are, they are, every year there are new tools coming up and pywbs tries to wrap them around and create a WPS or web processing service and develop around them simply and
Luckily, you can find more informations in a more structured way on the pywbs.org webpage. What is it not? Some people have too high expectations maybe or probably too different expectations.
It totally shouldn't be complicated for usage from the programmer's perspective. There is no client for web processing services. It's just pure server. I think more. No fancy web application No, even better desktop application, I think. Therefore, yeah, no graphical user interface apparently, so we were heavily inspired by map server and
There are no processes on the server. People are asking some from time to time does pywbs know how to buffer the responses? Like no, but write the process which will buffer vector data is like two lines of code.
Nobody did web processing service introduction yet. Very basically, it's one of the OGC XML based files you apparently have get capabilities
requests and then there are some other metadata requests like describe process and of course the most important one is the execute request which is calling the process instance or starting a process instance on the server and At the end of the day or maybe a next day you as a client should obtain some result of the calculation. There are three
predefined data types in the version most common version of the standard which is the literal type of data so basically text input or output complex data which is referring to vector data features as well as raster data files and
bounding box for specifying Yeah, left lower and top upper coordinates pair of coordinates of some region of interest Most used in the practice in practice on daily usage are literal data and complex data
I don't see much processes using bounding box Yeah This is how it works simply that you have a container with Supporting or offering many processes via the web processing service standard and the client
Communicates with a container or with the instant with the installation through the three type of request. I was just describing Yeah the essential functionalities of pywps is
as we said as I said is the communication communication bridge or maybe envelope between The tools and the Internet it fetches the required data from various data sources for for you So to say in a most safe way in a way that it takes care on how big the data are
And in case they are simply too big it will refuse to download them and to process them as well It creates containers for running processes process instance and tries to make sure that each container don't Interact with each other and that it behaves and on the most
Secure way as possible because you are at the end of the day executing code on your server So you really want to make sure that you have just executed you can't the code you allowed to be executed Of course, it takes fair under it takes care on the process management itself So communication communication between process between the core and the process itself
Reporting logging and other Yo, it's storing The data the final process data into some storage which can which can be a file maybe a database where the final
Datasets are stored saved and of course it Notifies is the client about process status and yeah progress. I Are major It started in 2006 and since then it was developed. We had multiple versions. I
Believe that I will just skip that so I just wanted to come to the recent version Because Sorry
Here we go Officially of pywps4. This is the current version Here we go And so today we are talking about pywps4. There was a big restructure realization of the code and We graduated last year as OS geo project. We were probably the longest project in the incubation process
It took us nine years. I always said it proudly and loudly it took by nine years to graduate to as Joe but now we are Well official OS geo project and happy member of the OS geo family and
What are the key features of pywps4? Because pywps4 was written from scratch from the previous version more or less more more than less Because of they were new technologies flying around on the internet
Which could and should be used as well, for example the L XML library which makes the parsing and Generating of XML Text files much more easier and faster as well We are using On heavily depending on flask as well
There is this WSGSI interface which had to be somehow yeah addressed A new multi-processing modules for parallel process calls Yeah, and of course Docker it's one of the technologies you totally probably need to address as well because of the containers and it's
It seems to be very promising technology In pywps4 we finally are Validating some data in the previous versions. We didn't care much. What did the client sense? Send is sending to the server as long as the data could be processed
by the underlying tool like GL for example or Grass GIS as long as it could be imported. We assume that the data are okay with the current version user though Yeah user of pywps can set for each process
Level of validation of the input data. So basic for each data type The basic is of course no validation and then there is also the the most strict one is very strict validation Which uses the XML scheme, I I guess that most of the
Implementations out there or most of the usages will just stick to strict validation which just attempts to Compare the mime types and probably and Make sure that it fits somehow that if do you if the client claims that there is a JSON file going in then?
It's probably a JSON file as well But if you need to be strict to and because of any reason you can go with XML scheme validation as well this of course applies just for GML inputs type of inputs if
You for some reason needs to send sri shapefile or gorgeous nor gml the validators need to be adjusted accordingly and Pywps is currently having a validator classes for these type of inputs the other ones would have to me Would have to be at
How our process containerized there is a lots of happening in the recent development the standard weight is that
The process the Pywps instance on every execute call creates a separated Directory and within this directory all the data or the input data are downloaded as well as or stored as well as Result of the execution is stored down as well with Pywps for we try to
We aim for safer approach like using docker container so that every process runs in separated docker environment and At the end of the day So that there is no there is no Danger that
The processes will simply rewrite each other's data stuff like that There is also a new mechanism of How the polarization is handled in the previous version the only way how you could
Steer the number of processes which are allowed to run in parallel Was just by a number number of processes which can simply run in parallel in the configuration file in this case There was a big issue if you want if you need it for some reason to have
If you had a probe if you had the process which would interpolate digital elevation model Which would be very much a computer. Sorry processor demanding as well as hard drive demanding and If there would be four calls in parallel and the number of processes allowed would be for example three
Then the last one would be just thrown away so to say it wouldn't be allowed allowed to run on the server With pywps4 We firstly about onto the usage of OS fork, which we didn't which wasn't working on on windows of course, but there is a
Mechanism which will store the requests and put them in the queue so and all the time some process well and the calculation it will just look in the Database whether there is another whereas there is whether there is another request for father
process execution and then it would just take The last call from the stack and continue with the process execution so in this way we can store 50 requests for example in the queue and So that the client
shouldn't get The Exception that there are too many processes on the server being executed much too often yeah, now the development what was happening recently is that we
Completely have returned back to template based XML Rendering Which is much faster and basically also very much easier to be implemented that would be probably the biggest change and This is where we are right now
We have what we would be like What would we like to do? implementation of wps2 0 it is around for a couple of years already Let's keep I don't know the details, but we are giving this talked on many phosphor g-cent every year last
In last couple of years. We are saying yeah, we would like to implement wps 2.0 I Personally doubt that we are going to do that We are probably skipping we are probably going to skip this and we are going to for the open API Directly, so let's call it wps3 According to the wfs 3 pattern, but the features which will be needed are
the features which Wps2 introduces will be needed anyway. It is like pausing of the processes Maybe deleting of the running processes stuff like that has to be implemented and we have to went best way how to do how to
Achieve this with Python docker containers flying all around. I was mentioning this several times we still I'm still personally not satisfied with the way how it this is done and I would like to incorporate docker instances more within the pywps core
one of my dreams which was in the version of the web processing service sorry in pywps 1 0 was that the output of the process Shouldn't be only a data file like for example geotiff for shapefile or gML
But it could be a service directly so that the client could make a request okay I want please process my My vector data and result should shouldn't be gML But the result should be wfs endpoint generating on the fly this was in the first version of pywps, and we didn't really implement it so far but
One of the ideas was that this would be implemented with help of Either map server or geo server preferably both of course as a back-end engines I? Was already mentioning geo API or open API sorry for
Yeah, my mistake There was a presentation in the previous section about it and pywps totally needs to be part of the family Our students of Google summer of cold Projects in recent years make some effort for the output storages
I was mentioning that usually the output goes to Directories or but but students were working heavily on having the have it a possibility to store the outputs to either Amazon S3 storage or maybe to post GIS database
Yeah support for the languages of course and other How is the project? organized We are OG OS geo project that means we have a project steering committee some of the members of the committee are here in this room like myself
Don't correlate is sitting behind Most of you. Thank you for coming and Luis Chairing the session in the other room If you want to join us if you want to join our community either go to getter
Which is probably the preferably the preferably place to have chat and to come in communicate as well as you can join us on the mailing list on OS geo and We are trying to attend sprints and conferences as for sport G What I personally say we are failing in is fundraising
Yeah We are incubated OS geo project that guarantees certain level of Sustainability and that we have a healthy and growing community. That means we
But it doesn't mean that we can give up and say okay. This is the given statues or granted we have to work on this heavily and Find new resources for the further development as well Wow, I have still three minutes. So very fastly how to get it up
install preferably with install This is how the how the Installation looks like and the most important part is By WPS configuration file along with processes directory where all the processes can be started as single Python files
This is the project structure. I'm sorry process structure a process called aria Is a class derived from process and it has list of inputs as well as list of outputs and
at the end there somewhere has to be a executing function Which obtains requests and response? and This is the code you should be able to write in a case You are a Python programmer and this should be relatively easy for you to understand. So what I see here. Oh
This is nasty. I would never do that. But if you if you totally need to do this if you need to transform Some file to the other format and use ogr2 ogr. This is possible, too So let's say this demonstrate the possibilities. It has not necessarily I would call this a production code
Very briefly about the philosophy still one minute Oh Louis loves idea of bikes. I love bike tools. So This tries to demonstrate that Pywps is like a bike apparently we have bike in the logo bike can be
By can have different shapes, but still can bring you from point a to 4 point B There can be large there can be small, of course They can be very cheap once but expensive and fancy ones There can be very simple and very complex. I have rather preferred a simple one in this case
If I would speak about pywps So this all describes our philosophy it should be versatile Pywps can be whatever you want it to be and it should be very much ease of use and easy to or self self-repair
You just need to know how to pedal Question time. Thank you. In case you have no question. I have some but please
So my questions well technical again, but how come you chose to use? standalone docker docker containers as opposed to Services because that way you are practically reduced using the resources on one system while if you would have services like for I know the little using docker swarm or Kubernetes you would not be
Restricted one system, but to a cluster which can be grown or grown or Shrink and automatically and also the number of services that are on automatically So it just seemed okay, perhaps it's not that easy, but it's much more easy to scale everything like that
Thank you for your comment, I'm not personally involved in this in this part of code So we will be welcome to join us and the mailing place latest I would like to see some demos as well from what I know one of the there. There are two places where you can
Put docker in or inside of pywps either you can containerize You can containerize the whole Instance or what we are trying to do is that each process instance starts its own docker container on demand Deploys it somewhere and this will be probably or point if I understand correctly and then we
Reuse the web processing service interface to call inside a process one more time Okay, I Was just saying that if you use it as a service it would be more scalable and
Possibly less resource resource hungry because you don't have to have already prepared something Basically, if you need something you'll be spawned if you don't need it. You will be closed down. So that's what I was Kind of okay. That's More technical. Thank you
Thanks for the comments as well and yeah, so imagine you run a process and you spawn a docker container or something If you run a process and you spawn a docker container and then your server crashes what happens to your processes
Can you actually get the results? When the question was if I understand correctly what if I run it in the container and the server crashes? Like the instance of the server. Yes I mean if say you send a request which takes two days to process and After two hours the web server crashes the BPS server crashes, but your process is still running
Can you get back to the process somehow? Any handle the server crashing and coming back up again, you know, no once it crashed the you should immediately obtain a
Execution exception report and Pi WPS tries as hard as possible to clean everything behind itself so that there are no data Lying around and no zombie processes running around same applies for the docker as far as I know these Docker was was one of the possibilities we were looking for by the way also for the process posing as well as process killing
So So one more time to repeat my answer if some process crashes We try to kill it and clean afterwards everything as hard as possible If it answers your question
Or if this isn't okay. Yeah, Jana has one Do you have a users list as well as a developers mailing list if I just want to use Pike WPS rather than developing it
Do you have a mailing list I can use? Sorry. Yeah, that was a easy one. No But but but yes But yes, they are they are we are we are not we there is not like
400 people at a mailing list You don't have to be afraid of two technical questions lying all around You just can join and pretend you are normal user and you will be kind To as well as if you join Gitter Then you might get some response from some of the community members in much nicer way Thank you
More questions still If it isn't the case, thank you for your attention and see you here or in other rooms in five minutes