Web Processing Services Using OS OpenData
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 95 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/15594 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Nottingham |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSS4G Nottingham 201393 / 95
17
25
29
31
32
34
48
50
56
58
68
69
70
82
89
91
00:00
Open sourceDemo (music)Computer fontLocal GroupSoftwareProcess modelingStochastic processParameter (computer programming)World Wide Web ConsortiumUsabilityForm (programming)Standard deviationImplementationCalculationEndliche ModelltheorieKeyboard shortcutInclusion mapAlgorithmWeb serviceClient (computing)Ultraviolet photoelectron spectroscopyWide area networkValue-added networkMaximum length sequenceInfinityGrass (card game)Mountain passAbstractionMaizeProof theoryMIDITwin primeUncertainty principleServer (computing)Theory of relativityMathematical analysisService-oriented architectureMathematicsMultiplication signLie groupStochastic processInformationProof theoryObservational studyWordStrategy gameEndliche ModelltheorieAreaArchaeological field surveyMereologyWeb 2.0Type theoryDependent and independent variablesContext awarenessOperator (mathematics)Group actionTelecommunicationPresentation of a groupCircleClient (computing)Standard deviationWeb browserBitSpeech synthesisSystem callLattice (order)WebcastLevel (video gaming)Arithmetic meanGoodness of fitElectronic program guideOpen sourceFactory (trading post)State of matterTwitterDigitizingCartesian coordinate systemOpen setSet (mathematics)Procedural programmingRaster graphicsVector spaceWater vaporMarkov chainActive contour modelUniform resource locatorDatabaseFunction (mathematics)Parameter (computer programming)Software developerProcess (computing)Computer fileComputer fontMetadataWebsiteComputer simulationFront and back endsLevel of measurementComputer animation
05:46
Dedekind cutComputer clusterOnline helpSpherical capStochastic processUncertainty principleGrass (card game)Level (video gaming)Personal area networkPhysical lawLocal area networkGotcha <Informatik>Content (media)EmulationFunction (mathematics)InfinityUltraviolet photoelectron spectroscopyContent management systemIRIS-TInclusion mapLibrary (computing)IntelRule of inferenceNormed vector spaceUniform resource nameProcess (computing)outputElectronic meeting systemComputer iconServer (computing)Numbering schemeCausalityClient (computing)Ext functorGeometrySoftwareSet (mathematics)Parameter (computer programming)Scripting languageStreaming mediaDirection (geometry)Coordinate systemWeb browserProcess (computing)Server (computing)Open sourceGrass (card game)outputService-oriented architectureStochastic processWeb 2.0Front and back endsObject (grammar)ArchitectureTerm (mathematics)Procedural programmingScripting language1 (number)Level (video gaming)Water vaporConfidence intervalSoftware frameworkCartesian coordinate systemMappingConnectivity (graph theory)ResultantPlug-in (computing)Analytic setQuantumMathematical analysisMobile appEndliche ModelltheorieCoordinate systemMereologyClient (computing)Point (geometry)IdentifiabilityGUI widgetDatabaseInternetworkingSoftwareFunction (mathematics)Software developerElectronic mailing listStandard deviationImplementationArchaeological field surveyOrder (biology)MassMultiplication signNeuroinformatikGreatest elementSocial classBoss CorporationWebsiteSymbol tableInsertion lossUniform resource locatorDisk read-and-write headMoment (mathematics)Local ringSurvival analysisFlow separationComputer animation
11:32
Metropolitan area networkInformation technology consultingWeightMoment of inertiaProcess (computing)AreaWeb 2.0Front and back endsProof theoryMappingCalculationOpen setMobile appPoint (geometry)Buffer solutionZoom lensFlow separationLevel (video gaming)Order (biology)Metropolitan area networkQuicksortUniverse (mathematics)Insertion lossIterationRule of inference
13:15
Information technology consultingDynamic random-access memoryWeightPoint (geometry)Online helpPersonal area networkGamma functionKnotCalculationPoint (geometry)FeedbackProcess (computing)Level (video gaming)Stochastic processSingle-precision floating-point formatWhiteboard
14:02
Gamma functionInformation technology consultingOnline helpArmFunction (mathematics)Server (computing)Information technology consultingNetwork topologyShape (magazine)WordComputer animation
14:23
Amsterdam Ordnance DatumArmInformation technology consultingSummierbarkeitOnline helpShape (magazine)Moment (mathematics)TunisProcess (computing)Validity (statistics)Operator (mathematics)Computer fileComputer animation
14:46
Open sourceOpen setMathematical analysisStandard deviationServer (computing)SoftwareProof theoryLevel (video gaming)Complex analysisBuildingEndliche ModelltheorieMilitary operationWorld Wide Web ConsortiumClient (computing)Standard deviationOpen sourceCalculationImage resolutionCuboidBitScripting languageSoftwareService-oriented architectureProof theoryOperator (mathematics)Endliche ModelltheorieImplementationTerm (mathematics)Server (computing)Computer simulationRegular graphAngular resolutionMarkov chainAreaGoodness of fitSoftware testingState of matterCondition numberMetreComputer animation
17:33
Information technology consultingShape (magazine)Computer-assisted translationMassImage resolutionPoint (geometry)Proof theoryOpen sourceSet (mathematics)Module (mathematics)Streaming mediaSoftwareDatabaseCASE <Informatik>TelecommunicationLogicUniform resource locatorPrototypeStandard deviationClient (computing)Computer fileServer (computing)Universe (mathematics)Dependent and independent variablesProcess (computing)Grass (card game)Endliche ModelltheorieKeyboard shortcutCartesian coordinate systemMoment (mathematics)Computer simulationFront and back endsFunction (mathematics)Scripting languageoutputBitAreaLine (geometry)Complex analysisPhysical systemScaling (geometry)Mathematical analysisMetreString (computer science)Insertion lossConnectivity (graph theory)Computer programmingQuicksortCalculationGroup actionCellular automatonMultiplication signMaizeSoftware testingSound effectSummierbarkeitStochastic processOrder (biology)Contingency tableElement (mathematics)Hand fanFood energyDemosceneState of matterResultantCausalityWordBasis <Mathematik>Bit rateInformation technology consultingMathematicsComputer animation
Transcript: English(auto-generated)
00:01
So, my name is Hugo Martins, I come from Wouter Consulting and the presentation is about using web processing services with ordinance survey open data Well, this is the outline of the presentation
00:21
We'll start explaining how we came up with the idea of building something which we call Catchment Finder and then we go on speaking what it is in fact and how it is working and in the end we'll draw some highlight conclusions We are specializing mainly in open source GIS
00:44
both desktop and web GIS, we do software development for QGIS and also web GIS bespoke applications and within the GIS world also we are having a niche specialization in numerical modeling concerning water engineering
01:04
and because we do a lot of water engineering we have commonly, we came up with the idea of building Catchment Finder because we commonly use a simple procedure that takes a bit of processing in a desktop GIS application
01:21
which is to calculate a river basin and we do that a lot for our modeling and it was something that we realized that it was happening lots of times and we had to process all the data and then make some quality analysis of this data and it was taking some time to tweak all the parameters in all the operations that we were making
01:43
So, we came up with this idea of building Catchment Finder which we use for this kind of simple processes and also as a proof of concept of web processing services So, the Catchment Finder aims mainly at combining all this so we pre-process the data that we took into a database
02:09
and then we make this process available through the web As I was saying, it doesn't mean that it needs to be used through the web browser although Catchment Finder itself is also a web GIS application
02:21
but the web processing service itself can be used also through the desktop So, what it means is that it doesn't matter if you are a GIS guy or not you can use it in a straight way in a straight way because you don't need to know how to do each step of the processing chain, let's say
02:46
So, first thing, we cannot run a process without data So, this was the first thing that we came up with Ordnance Survey is providing several open data sets and from those we took out Landform Panorama which is a data set which is having a vector layer with contours
03:09
and also a digital terrain model in ASCII GRI format, I suppose I'm not sure about that but I know it's a raster file and also we took the strategy data set which is just a vector with providing contextual information
03:24
like urban areas, streets and gazetteers So, we took that also just for putting some more information to give context in the web GIS application that I'm going to show you later This is on the Ordnance Survey website for open data
03:45
So, you can download all this data, it's open and not only you can freely get it but you also can use it for making your kind of studies, analyses and make some outputs of it So, it's open data So, now we have the data, we have to find a way of providing this service that we have in mind
04:06
So, this chained model of GIS operations that allow one to calculate a river basin So, what to use? Well, we obviously wanted to use OGC standards and for that there is one which is called WPS which stands for Web Processing Service
04:23
and mainly this standard is specifying a way of communication between client and server that allows one to not only find processes that are available in the server but execute this process in a way that is typically done in a GIS desktop application
04:42
but through the web So, the great advantage of it is that you can have really, really from simple things to really complex models in the back end and you just need to give the inputs and you'll receive the outputs and you don't need to make all the operations in between
05:00
the process will take care of it So, that's the big advantage of it How is it working then? So, the standard is specifying three types of requests So, each time the client sends a request to the server the server understands this request which can be get capabilities, describe, process or execute and then it sends back the answer to the client
05:24
All these requests, there are three, so get capabilities the request looks like the URL that I'm showing you and it returns like an XML response which is basically describing service metadata so it's describing, giving general information to the user
05:43
who is providing the service and what kind of processes are available in the server, for example, you see here in this lower part it says process offering so there it describes how many web processing servers are available in this server
06:01
So, once you know which processing services you have you want to know what they are doing exactly and that's why there is describe process, which is another request where then you use a simple identifier that you got from the previous offering list so you say, oh, okay, I see that there is a processing services that is called SimpleGrass
06:22
but what does it do? and that's why you use this request and then you can see that it has a title, an abstract and also it's telling you which inputs it needs to take so that it can run on the server side and also it's describing what are the outputs of this process
06:41
so you can see here in the lower part that you have at least three outputs from this process Now, you know that you have a process that you want to run you know which inputs you need to give to the process so now you can execute it
07:00
and, for example, for that process that I was showing sorry, I've just come back so you can see here that it's taking an X coordinate and it would also take a Y coordinate, I just hide it because it was too much and that's what we are specifying in the execute process
07:20
we give back to the server an X coordinate and a Y coordinate and then from that point the process will calculate the catchment itself so this is how you work with the standard so, yeah, now we have the standard okay, we have the data
07:40
what are we going to use to provide this service so we had to choose an architecture to implement our objective and obviously open source was the way to go forward and we had in mind to build the server side and also a client side
08:01
so for the server side we were using already GRASS we have lots of experience with GRASS we like GRASS, GRASS is one of the first open source desktop GIS desktop applications and, yeah, we were doing lots of processes with it so we decided, yeah, if we can find something that works with GRASS in the back end
08:23
it would be nice because then we have already all these procedures that are already implemented within our framework of water engineering modeling and so we went to search some kind of server that would communicate with GRASS and provide the web processing service
08:40
and that was easy to find, it was PyWPS which is providing native support for GRASS and R obviously PyWPS is implemented in Python so you can use several other packages that are working in Python so if you want to use Shapley or GDAL or whatever kind of package that is in Python
09:00
running in Python, you can call it through the PyWPS so it seemed a nice framework and it was also giving a lot of confidence because it was also one of the first implementations of the WPS standard in the open source world so we went for this PyWPS software and PyWPS is also depending on map server
09:21
because the standard defines that you can send back to the client numbers, letters, strings, but also maps like WFS and WMS results of your process so PyWPS is also connecting to map server to send back WMS or WFS data to the client
09:46
then we wanted to build a customized client on the web although this server component is working for example you can run this process in Quantum GIS it has a plugin which is called WPS something you can install this plugin and then you can run this process from Quantum GIS
10:02
but we wanted to make like a custom application on the web so we decided to use OpenLayers I think everyone knows about it it's a JavaScript framework awesome JavaScript framework that allows you to build spatial apps in the web then we decided also to go with EXTJS it's a big, big framework that allows you to develop rich internet applications
10:23
that are more or less minifying the desktop applications like Violex which is just a middleware framework between those two that are already making some widgets which combine both EXT and OpenLayers
10:40
well, so how did we start the development? first thing to do was obviously include all the data inside the GRASS database so we set up a location, a map set for the ones that know how to work with GRASS these are always the first procedures then we imported the data and made some quality analysis on it
11:00
and finally, because we had all the procedure in terms of analytical procedure we had already developed so we just converted this procedure into a Python script that would be called by PyWPS and, well, we converted to the Python
11:20
then we used it in PyWPS and everything was set up to test this this web processing service so I'm going to do a quick demonstration hopefully it works I think it should work so this is really simple it's just a proof of concept it's just a really simple web GIS
11:42
well, you see that we used open data for conceptualization and just putting some nice maps in the background and to see how easy it is and for sure this process is taking three or four middle steps to come to the end so that you can build so normally it's used only by GIS people
12:03
because they know the inner workings of this processing but with this kind of apps the user doesn't need to know what is happening behind he just knows that oh, I want to calculate the river basin so, okay, the first thing that he needs to do is just defining a point
12:22
in the map and then calculate accident or catchment, sorry okay, catchments in that area yeah, that will be driven from this point so there's like a buffer around your X, Y now, in the process itself what we do is that we try to find
12:40
the closest point in the river to this point that was user defined and so from the river you can calculate the catchment oh, okay so he knows to go, okay because, well, you can put the point in several places but then because we can snap it to the river and find the closest point in the river
13:00
then we can calculate the catchment for that line, okay for that river so that's what's happening in the back end so the user doesn't know anything about this so now the process was calculated and, yeah, we'll zoom into it so it was really small as you can see well, what was the problem here
13:21
I can explain to you it's because it's really flat so I'll show you one so that you can see that it's working I was unlucky with the point well, hopefully I have well, let's put some point over here hopefully
13:40
calculate catchment yeah so you see that because the process is running asynchronously the user is kept always in constant feedback so he knows where it is at each stage so you can see it's now calculating and, yeah, there you go
14:01
then we have a catchment there as you see, this output was coming from the server as a WFS and now we can also, using PyWPS we can also retrieve a shapefile to the user so we decided to give that possibility
14:21
so the user can just download it and there you go and, yeah, inside of the zip file we will have the catchment shapefile like that so it was quite easy for each kind of user it doesn't need to be a GIS guy to make this process and it doesn't need to care about the data or about which operations to do
14:42
about which validations to do everything is happening in the backend side so, in terms of conclusions what we can say is that well, WPS standard is really allowing one to develop really complex models
15:02
and turning it into a really transparent tool into the user I mean, if we all GIS guys know that we can do really simple things to really complex stuff and we can chain lots of operations together and if we give those tools to a regular user which is not a GIS guy
15:21
it doesn't understand anything about what it's doing but, using WPS what we can do is that we can say, okay, this tool is specifically to do this and it doesn't matter if it's just one operation or 30 operations it's just built on the backend side and the user doesn't need to care about the data and about how to make the workflow
15:42
so, this WPS standard in fact, was developed by GIS people to GIS people, but it can be used with anything even just calculations, you know regular calculations and numerical modelling it doesn't really need to work with geospatial data but it's mainly directed to
16:01
or work with geospatial data another thing to point out was that all the implementation in open source software is quite stable and really, really robust so you can work with it and it just works, you know if you have a little bit of knowledge and try a bit
16:21
it just works, almost out of the box obviously not the Python script that's something that you need to do by yourself because the server doesn't know what you want to do so, you need to do it by itself and, you know, another thing is that without open data we wouldn't be able to make this proof of concept
16:40
because, yeah, it's not only that the data is open free, and you can get it it's that you can also provide services with it so, the only problem and you saw that in the first catchment is that this data, in terms of spatial resolution is not that high spatial resolution that's why sometimes you get really small catchments so in areas that are more flat
17:03
and everything is flat it's difficult to have a good catchment calculation so it would be nice to have higher resolution data sets so that we could, you know, provide a service that is a little bit more accurate but this is just depending on the underlying data
17:21
and, yeah, I think that is it Any questions?
17:42
I think it's 50 meters, I think so Yeah, in fact we are at this moment
18:01
developing an application which is happening, which is working like that we have four data sets and we allow the user to upload his own so we can get higher resolutions of it and then the process, instead of working with our base data it starts to work with the user data so it's possible through the WPS process itself
18:22
so it's, yeah, in fact we are doing it right now Just quickly following on from that does that mean you point the WPS process to their source data or do you actually first copy it up and then point it to your local data? We have a, so for the standard one
18:42
in this case we have pre-processed all the data and it's already there living in the GRASS database so when we run the process I don't need to import everything so it's just faster because it's living there but if the user uploads its own files then I need to import it into the GRASS database
19:01
and then I can work with it as I would work with any kind of data set Because the input to your WPS process is something specific to your GRASS database You can't say, right, I have a terrain server here I can give you DTM I just want to point to this At this point no, but it really depends
19:21
on how you implement the WPS itself because you are coding it so if in your WPS process you say that one of the inputs is the URL of another service that is providing you data then you need to code that logic in the WPS script to go and fetch this data and then you can work with it
19:41
So it's quite flexible In fact the WPS standard itself is not implementing the process of managing the way of communication between client and server but it's really flexible because it's Python so it's really easy to develop things It doesn't take ages
20:01
It might be discussable about the performance Until now from what I've done with PyWPS the performance is great But Python is just nice to do quick prototyping and to implement this kind of service So it's really up to you what you are expecting It could be a new URL It could be an integer, a string
20:22
It can even be a GML feature or WMS server, whatever and then you make the logic in the backend script in the server
20:45
Not the GRASS itself but PyWPS is the server-side component that is dealing automatically with setting up the GRASS
21:00
the communication between GRASS and the PyWPS itself So you can call any other program that has Python bindings So for example this shapefile to be honest was immediately outputted from GRASS as a shapefile but I could use it with OGR because OGR also has Python bindings so we can call it in GDAL
21:21
and SpatialLite Even PostGIS you can access things through the WPS to PostGIS, insert in the database make analysis in the database retrieve it to the PyWPS and then you say okay I want this output as a WMS or as WFS and it's just really really flexible
21:41
You can do whatever you want It just depends on how you call it In like an urban area I would probably have to have higher resolution data Yeah, yeah I'm going to be looking at
22:01
like vacant lots, sloughs just anywhere I might be able to find the catchment and you have to individually point on it to do the calculation to find it So you would have to sort of like sort of micro-task the process and say I'm going to look in this quadrant
22:22
this quadrant this quadrant Is that how you would do it? No, no, this is a simple process It's just a proof of concept You can put it as much complexity as you want but this one is just you put a point there and the process itself in the back end is well this point is not in a river line
22:42
so I'm going to find the closest point in the river So if you are putting it in an urban area where there is no stream it will calculate the closest point that is snapping into the river the closest river So if you want to do it in urban areas
23:01
or I don't know it's just you need to put a little bit of depth because it's just 50 meters of resolution so it's not that great to be honest and well at the national scale it's nice at the national scale and also it's just giving you some outputs but if you want to really work
23:20
in smaller areas and especially where they are flat you need higher resolution data but it's up to you if you have this data or even like they were asking before if you want to provide the user it's possible it's really possible
23:44
Yeah? Do you have to put in grass to run it or is it possible also to link to the system and which kind of which kind of model It can be a hydrological model or another way to Well, you don't need to put it in grass
24:02
if you can call it from Python Okay, so if you have Python if it's something like a C module and then you have the Python bindings to it then you can call it directly otherwise for example we had something like this we had to build a new grass module so we made a C module
24:21
a new module and we have put it into grass because it would be faster than in Python so it was for performance issues but we could have done it with Python and you know there is this nice add-on to Python module which is NumPy and SciPy and this is like providing features really similar to something
24:40
that is widely known a proprietary software and that is for numerical modeling so you can work with this it really depends on the things that you know how to do and it depends if you well, if you are worried or not about performance issues so in this case for example we have this complex model
25:01
that we tried it in Python because it was fast so we made a new C module for grass and grass is working with that sorry? it's called NetWet we didn't release it yet
25:20
because to be honest the module itself was not done by us so it's from the university but we are speaking with them and probably it will be included in future releases of grass as a new module you know r.netwet so this one is only attached for river?
25:43
yeah, yeah, yeah it's just for that main purpose it's just like this we wanted just to make a proof of concept because WPS is a really nice standard but it's not being used as much as we were expecting but you can do it you know even for simple things it's just a nice way to communicate with the server
26:01
and it's really straightforward then you have all these and you find the server understands what you are asking and the client knows what to ask and how to ask it and how to read the response from the server so it's just a nice standard to then you can abstract yourself from this reading, inputs, outputs you just need to worry with the process itself
26:22
so it's just that's why we did it just as a proof of concept but it could be much more complex thank you thank you