We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Web Processing Services Using OS OpenData

00:00

Formal Metadata

Title
Web Processing Services Using OS OpenData
Title of Series
Number of Parts
95
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language
Production PlaceNottingham

Content Metadata

Subject Area
Genre
Abstract
In April 2010, Ordnance Survey made a number of their national mapping products freely available under the OS OpenData initiative. Vector and raster datasets at varying scales were released under a very permissive license which allows users to freely create derivative works, even for commercial purposes. Lutra Consulting released a WebGIS application to demonstrate the value and potential of combining OS OpenData, OGC services & standards and open source GIS software. The WebGIS application, Catchment Finder, uses the OGC Web Processing Service (WPS) to provide a simple method for users to generate hydrological catchments (or watersheds) for any point in the UK. Catchment delineation is based on the OpenData Landform PANORAMA dataset, a 50 metre resolution digital terrain model (DTM). Catchment Finder was developed using the following FOSS components: OpenLayers and Ext JS for all user-facing functionality. MapServer and TileCache to serve background mapping and processed results. GRASS GIS for server-side catchment delineation process. PyWPS to provide a mechanism for interaction between the browser and GIS processing taking place on the server. GRASS GIS sits at the core of Catchment Finder. National slope and aspect raster datasets were pre-calculated as inputs for the watershed analysis module in order to optimise calculation times. A WPS process was developed in python (using PyWPS and GRASS’ python bindings). The process chains together a number of GRASS commands in order to generate a vector layer representing the catchment outline which is then displayed in the web client via GML or optionally downloaded as a Shapefile. PyWPS (based on python) was chosen in preference to alternative WPS server implementations due to the typical flexibility and efficiency offered by python (a high-level programming language). Implementing specific GIS processing tasks as WebGIS applications simplifies the end-user’s tasks and therefore opens up GIS processes to non-technical people. Storing datasets and carrying out processing centrally helps remove the burden of managing large/national datasets. Any updates to underlying datasets can be carried out centrally with minimal impact. As Catchment Finder implements the OGC WPS standard, it is also possible for the service to be utilised by desktop GIS applications. At present, due to the low resolution of the underlying DTM, it is only possible to generate watersheds for larger watercourses.
Open sourceDemo (music)Computer fontLocal GroupSoftwareProcess modelingStochastic processParameter (computer programming)World Wide Web ConsortiumUsabilityForm (programming)Standard deviationImplementationCalculationEndliche ModelltheorieKeyboard shortcutInclusion mapAlgorithmWeb serviceClient (computing)Ultraviolet photoelectron spectroscopyWide area networkValue-added networkMaximum length sequenceInfinityGrass (card game)Mountain passAbstractionMaizeProof theoryMIDITwin primeUncertainty principleServer (computing)Theory of relativityMathematical analysisService-oriented architectureMathematicsMultiplication signLie groupStochastic processInformationProof theoryObservational studyWordStrategy gameEndliche ModelltheorieAreaArchaeological field surveyMereologyWeb 2.0Type theoryDependent and independent variablesContext awarenessOperator (mathematics)Group actionTelecommunicationPresentation of a groupCircleClient (computing)Standard deviationWeb browserBitSpeech synthesisSystem callLattice (order)WebcastLevel (video gaming)Arithmetic meanGoodness of fitElectronic program guideOpen sourceFactory (trading post)State of matterTwitterDigitizingCartesian coordinate systemOpen setSet (mathematics)Procedural programmingRaster graphicsVector spaceWater vaporMarkov chainActive contour modelUniform resource locatorDatabaseFunction (mathematics)Parameter (computer programming)Software developerProcess (computing)Computer fileComputer fontMetadataWebsiteComputer simulationFront and back endsLevel of measurementComputer animation
Dedekind cutComputer clusterOnline helpSpherical capStochastic processUncertainty principleGrass (card game)Level (video gaming)Personal area networkPhysical lawLocal area networkGotcha <Informatik>Content (media)EmulationFunction (mathematics)InfinityUltraviolet photoelectron spectroscopyContent management systemIRIS-TInclusion mapLibrary (computing)IntelRule of inferenceNormed vector spaceUniform resource nameProcess (computing)outputElectronic meeting systemComputer iconServer (computing)Numbering schemeCausalityClient (computing)Ext functorGeometrySoftwareSet (mathematics)Parameter (computer programming)Scripting languageStreaming mediaDirection (geometry)Coordinate systemWeb browserProcess (computing)Server (computing)Open sourceGrass (card game)outputService-oriented architectureStochastic processWeb 2.0Front and back endsObject (grammar)ArchitectureTerm (mathematics)Procedural programmingScripting language1 (number)Level (video gaming)Water vaporConfidence intervalSoftware frameworkCartesian coordinate systemMappingConnectivity (graph theory)ResultantPlug-in (computing)Analytic setQuantumMathematical analysisMobile appEndliche ModelltheorieCoordinate systemMereologyClient (computing)Point (geometry)IdentifiabilityGUI widgetDatabaseInternetworkingSoftwareFunction (mathematics)Software developerElectronic mailing listStandard deviationImplementationArchaeological field surveyOrder (biology)MassMultiplication signNeuroinformatikGreatest elementSocial classBoss CorporationWebsiteSymbol tableInsertion lossUniform resource locatorDisk read-and-write headMoment (mathematics)Local ringSurvival analysisFlow separationComputer animation
Metropolitan area networkInformation technology consultingWeightMoment of inertiaProcess (computing)AreaWeb 2.0Front and back endsProof theoryMappingCalculationOpen setMobile appPoint (geometry)Buffer solutionZoom lensFlow separationLevel (video gaming)Order (biology)Metropolitan area networkQuicksortUniverse (mathematics)Insertion lossIterationRule of inference
Information technology consultingDynamic random-access memoryWeightPoint (geometry)Online helpPersonal area networkGamma functionKnotCalculationPoint (geometry)FeedbackProcess (computing)Level (video gaming)Stochastic processSingle-precision floating-point formatWhiteboard
Gamma functionInformation technology consultingOnline helpArmFunction (mathematics)Server (computing)Information technology consultingNetwork topologyShape (magazine)WordComputer animation
Amsterdam Ordnance DatumArmInformation technology consultingSummierbarkeitOnline helpShape (magazine)Moment (mathematics)TunisProcess (computing)Validity (statistics)Operator (mathematics)Computer fileComputer animation
Open sourceOpen setMathematical analysisStandard deviationServer (computing)SoftwareProof theoryLevel (video gaming)Complex analysisBuildingEndliche ModelltheorieMilitary operationWorld Wide Web ConsortiumClient (computing)Standard deviationOpen sourceCalculationImage resolutionCuboidBitScripting languageSoftwareService-oriented architectureProof theoryOperator (mathematics)Endliche ModelltheorieImplementationTerm (mathematics)Server (computing)Computer simulationRegular graphAngular resolutionMarkov chainAreaGoodness of fitSoftware testingState of matterCondition numberMetreComputer animation
Information technology consultingShape (magazine)Computer-assisted translationMassImage resolutionPoint (geometry)Proof theoryOpen sourceSet (mathematics)Module (mathematics)Streaming mediaSoftwareDatabaseCASE <Informatik>TelecommunicationLogicUniform resource locatorPrototypeStandard deviationClient (computing)Computer fileServer (computing)Universe (mathematics)Dependent and independent variablesProcess (computing)Grass (card game)Endliche ModelltheorieKeyboard shortcutCartesian coordinate systemMoment (mathematics)Computer simulationFront and back endsFunction (mathematics)Scripting languageoutputBitAreaLine (geometry)Complex analysisPhysical systemScaling (geometry)Mathematical analysisMetreString (computer science)Insertion lossConnectivity (graph theory)Computer programmingQuicksortCalculationGroup actionCellular automatonMultiplication signMaizeSoftware testingSound effectSummierbarkeitStochastic processOrder (biology)Contingency tableElement (mathematics)Hand fanFood energyDemosceneState of matterResultantCausalityWordBasis <Mathematik>Bit rateInformation technology consultingMathematicsComputer animation
Transcript: English(auto-generated)
So, my name is Hugo Martins, I come from Wouter Consulting and the presentation is about using web processing services with ordinance survey open data Well, this is the outline of the presentation
We'll start explaining how we came up with the idea of building something which we call Catchment Finder and then we go on speaking what it is in fact and how it is working and in the end we'll draw some highlight conclusions We are specializing mainly in open source GIS
both desktop and web GIS, we do software development for QGIS and also web GIS bespoke applications and within the GIS world also we are having a niche specialization in numerical modeling concerning water engineering
and because we do a lot of water engineering we have commonly, we came up with the idea of building Catchment Finder because we commonly use a simple procedure that takes a bit of processing in a desktop GIS application
which is to calculate a river basin and we do that a lot for our modeling and it was something that we realized that it was happening lots of times and we had to process all the data and then make some quality analysis of this data and it was taking some time to tweak all the parameters in all the operations that we were making
So, we came up with this idea of building Catchment Finder which we use for this kind of simple processes and also as a proof of concept of web processing services So, the Catchment Finder aims mainly at combining all this so we pre-process the data that we took into a database
and then we make this process available through the web As I was saying, it doesn't mean that it needs to be used through the web browser although Catchment Finder itself is also a web GIS application
but the web processing service itself can be used also through the desktop So, what it means is that it doesn't matter if you are a GIS guy or not you can use it in a straight way in a straight way because you don't need to know how to do each step of the processing chain, let's say
So, first thing, we cannot run a process without data So, this was the first thing that we came up with Ordnance Survey is providing several open data sets and from those we took out Landform Panorama which is a data set which is having a vector layer with contours
and also a digital terrain model in ASCII GRI format, I suppose I'm not sure about that but I know it's a raster file and also we took the strategy data set which is just a vector with providing contextual information
like urban areas, streets and gazetteers So, we took that also just for putting some more information to give context in the web GIS application that I'm going to show you later This is on the Ordnance Survey website for open data
So, you can download all this data, it's open and not only you can freely get it but you also can use it for making your kind of studies, analyses and make some outputs of it So, it's open data So, now we have the data, we have to find a way of providing this service that we have in mind
So, this chained model of GIS operations that allow one to calculate a river basin So, what to use? Well, we obviously wanted to use OGC standards and for that there is one which is called WPS which stands for Web Processing Service
and mainly this standard is specifying a way of communication between client and server that allows one to not only find processes that are available in the server but execute this process in a way that is typically done in a GIS desktop application
but through the web So, the great advantage of it is that you can have really, really from simple things to really complex models in the back end and you just need to give the inputs and you'll receive the outputs and you don't need to make all the operations in between
the process will take care of it So, that's the big advantage of it How is it working then? So, the standard is specifying three types of requests So, each time the client sends a request to the server the server understands this request which can be get capabilities, describe, process or execute and then it sends back the answer to the client
All these requests, there are three, so get capabilities the request looks like the URL that I'm showing you and it returns like an XML response which is basically describing service metadata so it's describing, giving general information to the user
who is providing the service and what kind of processes are available in the server, for example, you see here in this lower part it says process offering so there it describes how many web processing servers are available in this server
So, once you know which processing services you have you want to know what they are doing exactly and that's why there is describe process, which is another request where then you use a simple identifier that you got from the previous offering list so you say, oh, okay, I see that there is a processing services that is called SimpleGrass
but what does it do? and that's why you use this request and then you can see that it has a title, an abstract and also it's telling you which inputs it needs to take so that it can run on the server side and also it's describing what are the outputs of this process
so you can see here in the lower part that you have at least three outputs from this process Now, you know that you have a process that you want to run you know which inputs you need to give to the process so now you can execute it
and, for example, for that process that I was showing sorry, I've just come back so you can see here that it's taking an X coordinate and it would also take a Y coordinate, I just hide it because it was too much and that's what we are specifying in the execute process
we give back to the server an X coordinate and a Y coordinate and then from that point the process will calculate the catchment itself so this is how you work with the standard so, yeah, now we have the standard okay, we have the data
what are we going to use to provide this service so we had to choose an architecture to implement our objective and obviously open source was the way to go forward and we had in mind to build the server side and also a client side
so for the server side we were using already GRASS we have lots of experience with GRASS we like GRASS, GRASS is one of the first open source desktop GIS desktop applications and, yeah, we were doing lots of processes with it so we decided, yeah, if we can find something that works with GRASS in the back end
it would be nice because then we have already all these procedures that are already implemented within our framework of water engineering modeling and so we went to search some kind of server that would communicate with GRASS and provide the web processing service
and that was easy to find, it was PyWPS which is providing native support for GRASS and R obviously PyWPS is implemented in Python so you can use several other packages that are working in Python so if you want to use Shapley or GDAL or whatever kind of package that is in Python
running in Python, you can call it through the PyWPS so it seemed a nice framework and it was also giving a lot of confidence because it was also one of the first implementations of the WPS standard in the open source world so we went for this PyWPS software and PyWPS is also depending on map server
because the standard defines that you can send back to the client numbers, letters, strings, but also maps like WFS and WMS results of your process so PyWPS is also connecting to map server to send back WMS or WFS data to the client
then we wanted to build a customized client on the web although this server component is working for example you can run this process in Quantum GIS it has a plugin which is called WPS something you can install this plugin and then you can run this process from Quantum GIS
but we wanted to make like a custom application on the web so we decided to use OpenLayers I think everyone knows about it it's a JavaScript framework awesome JavaScript framework that allows you to build spatial apps in the web then we decided also to go with EXTJS it's a big, big framework that allows you to develop rich internet applications
that are more or less minifying the desktop applications like Violex which is just a middleware framework between those two that are already making some widgets which combine both EXT and OpenLayers
well, so how did we start the development? first thing to do was obviously include all the data inside the GRASS database so we set up a location, a map set for the ones that know how to work with GRASS these are always the first procedures then we imported the data and made some quality analysis on it
and finally, because we had all the procedure in terms of analytical procedure we had already developed so we just converted this procedure into a Python script that would be called by PyWPS and, well, we converted to the Python
then we used it in PyWPS and everything was set up to test this this web processing service so I'm going to do a quick demonstration hopefully it works I think it should work so this is really simple it's just a proof of concept it's just a really simple web GIS
well, you see that we used open data for conceptualization and just putting some nice maps in the background and to see how easy it is and for sure this process is taking three or four middle steps to come to the end so that you can build so normally it's used only by GIS people
because they know the inner workings of this processing but with this kind of apps the user doesn't need to know what is happening behind he just knows that oh, I want to calculate the river basin so, okay, the first thing that he needs to do is just defining a point
in the map and then calculate accident or catchment, sorry okay, catchments in that area yeah, that will be driven from this point so there's like a buffer around your X, Y now, in the process itself what we do is that we try to find
the closest point in the river to this point that was user defined and so from the river you can calculate the catchment oh, okay so he knows to go, okay because, well, you can put the point in several places but then because we can snap it to the river and find the closest point in the river
then we can calculate the catchment for that line, okay for that river so that's what's happening in the back end so the user doesn't know anything about this so now the process was calculated and, yeah, we'll zoom into it so it was really small as you can see well, what was the problem here
I can explain to you it's because it's really flat so I'll show you one so that you can see that it's working I was unlucky with the point well, hopefully I have well, let's put some point over here hopefully
calculate catchment yeah so you see that because the process is running asynchronously the user is kept always in constant feedback so he knows where it is at each stage so you can see it's now calculating and, yeah, there you go
then we have a catchment there as you see, this output was coming from the server as a WFS and now we can also, using PyWPS we can also retrieve a shapefile to the user so we decided to give that possibility
so the user can just download it and there you go and, yeah, inside of the zip file we will have the catchment shapefile like that so it was quite easy for each kind of user it doesn't need to be a GIS guy to make this process and it doesn't need to care about the data or about which operations to do
about which validations to do everything is happening in the backend side so, in terms of conclusions what we can say is that well, WPS standard is really allowing one to develop really complex models
and turning it into a really transparent tool into the user I mean, if we all GIS guys know that we can do really simple things to really complex stuff and we can chain lots of operations together and if we give those tools to a regular user which is not a GIS guy
it doesn't understand anything about what it's doing but, using WPS what we can do is that we can say, okay, this tool is specifically to do this and it doesn't matter if it's just one operation or 30 operations it's just built on the backend side and the user doesn't need to care about the data and about how to make the workflow
so, this WPS standard in fact, was developed by GIS people to GIS people, but it can be used with anything even just calculations, you know regular calculations and numerical modelling it doesn't really need to work with geospatial data but it's mainly directed to
or work with geospatial data another thing to point out was that all the implementation in open source software is quite stable and really, really robust so you can work with it and it just works, you know if you have a little bit of knowledge and try a bit
it just works, almost out of the box obviously not the Python script that's something that you need to do by yourself because the server doesn't know what you want to do so, you need to do it by itself and, you know, another thing is that without open data we wouldn't be able to make this proof of concept
because, yeah, it's not only that the data is open free, and you can get it it's that you can also provide services with it so, the only problem and you saw that in the first catchment is that this data, in terms of spatial resolution is not that high spatial resolution that's why sometimes you get really small catchments so in areas that are more flat
and everything is flat it's difficult to have a good catchment calculation so it would be nice to have higher resolution data sets so that we could, you know, provide a service that is a little bit more accurate but this is just depending on the underlying data
and, yeah, I think that is it Any questions?
I think it's 50 meters, I think so Yeah, in fact we are at this moment
developing an application which is happening, which is working like that we have four data sets and we allow the user to upload his own so we can get higher resolutions of it and then the process, instead of working with our base data it starts to work with the user data so it's possible through the WPS process itself
so it's, yeah, in fact we are doing it right now Just quickly following on from that does that mean you point the WPS process to their source data or do you actually first copy it up and then point it to your local data? We have a, so for the standard one
in this case we have pre-processed all the data and it's already there living in the GRASS database so when we run the process I don't need to import everything so it's just faster because it's living there but if the user uploads its own files then I need to import it into the GRASS database
and then I can work with it as I would work with any kind of data set Because the input to your WPS process is something specific to your GRASS database You can't say, right, I have a terrain server here I can give you DTM I just want to point to this At this point no, but it really depends
on how you implement the WPS itself because you are coding it so if in your WPS process you say that one of the inputs is the URL of another service that is providing you data then you need to code that logic in the WPS script to go and fetch this data and then you can work with it
So it's quite flexible In fact the WPS standard itself is not implementing the process of managing the way of communication between client and server but it's really flexible because it's Python so it's really easy to develop things It doesn't take ages
It might be discussable about the performance Until now from what I've done with PyWPS the performance is great But Python is just nice to do quick prototyping and to implement this kind of service So it's really up to you what you are expecting It could be a new URL It could be an integer, a string
It can even be a GML feature or WMS server, whatever and then you make the logic in the backend script in the server
Not the GRASS itself but PyWPS is the server-side component that is dealing automatically with setting up the GRASS
the communication between GRASS and the PyWPS itself So you can call any other program that has Python bindings So for example this shapefile to be honest was immediately outputted from GRASS as a shapefile but I could use it with OGR because OGR also has Python bindings so we can call it in GDAL
and SpatialLite Even PostGIS you can access things through the WPS to PostGIS, insert in the database make analysis in the database retrieve it to the PyWPS and then you say okay I want this output as a WMS or as WFS and it's just really really flexible
You can do whatever you want It just depends on how you call it In like an urban area I would probably have to have higher resolution data Yeah, yeah I'm going to be looking at
like vacant lots, sloughs just anywhere I might be able to find the catchment and you have to individually point on it to do the calculation to find it So you would have to sort of like sort of micro-task the process and say I'm going to look in this quadrant
this quadrant this quadrant Is that how you would do it? No, no, this is a simple process It's just a proof of concept You can put it as much complexity as you want but this one is just you put a point there and the process itself in the back end is well this point is not in a river line
so I'm going to find the closest point in the river So if you are putting it in an urban area where there is no stream it will calculate the closest point that is snapping into the river the closest river So if you want to do it in urban areas
or I don't know it's just you need to put a little bit of depth because it's just 50 meters of resolution so it's not that great to be honest and well at the national scale it's nice at the national scale and also it's just giving you some outputs but if you want to really work
in smaller areas and especially where they are flat you need higher resolution data but it's up to you if you have this data or even like they were asking before if you want to provide the user it's possible it's really possible
Yeah? Do you have to put in grass to run it or is it possible also to link to the system and which kind of which kind of model It can be a hydrological model or another way to Well, you don't need to put it in grass
if you can call it from Python Okay, so if you have Python if it's something like a C module and then you have the Python bindings to it then you can call it directly otherwise for example we had something like this we had to build a new grass module so we made a C module
a new module and we have put it into grass because it would be faster than in Python so it was for performance issues but we could have done it with Python and you know there is this nice add-on to Python module which is NumPy and SciPy and this is like providing features really similar to something
that is widely known a proprietary software and that is for numerical modeling so you can work with this it really depends on the things that you know how to do and it depends if you well, if you are worried or not about performance issues so in this case for example we have this complex model
that we tried it in Python because it was fast so we made a new C module for grass and grass is working with that sorry? it's called NetWet we didn't release it yet
because to be honest the module itself was not done by us so it's from the university but we are speaking with them and probably it will be included in future releases of grass as a new module you know r.netwet so this one is only attached for river?
yeah, yeah, yeah it's just for that main purpose it's just like this we wanted just to make a proof of concept because WPS is a really nice standard but it's not being used as much as we were expecting but you can do it you know even for simple things it's just a nice way to communicate with the server
and it's really straightforward then you have all these and you find the server understands what you are asking and the client knows what to ask and how to ask it and how to read the response from the server so it's just a nice standard to then you can abstract yourself from this reading, inputs, outputs you just need to worry with the process itself
so it's just that's why we did it just as a proof of concept but it could be much more complex thank you thank you