Using the OGC Web Processing Service (WPS) to move business logic to the server
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43319 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Thermodynamischer ProzessWeb serviceLogicServer (computing)Process (computing)EmailAddress spaceTwitterElectronic mailing listComa BerenicesSlide ruleLecture/ConferenceComputer animation
00:25
Web serviceThermodynamischer ProzessServer (computing)LogicPoint (geometry)WebsiteBitWeb 2.0Slide ruleServer (computing)LogicTextsystemProduct (business)Level (video gaming)Thermodynamischer ProzessMappingKey (cryptography)Information2 (number)Thomas BayesState of matterComputer animation
01:21
PolygonBit rateComplex (psychology)BendingMaizeScripting languageJava appletQuicksortAreaDatabaseOpen setPolygonWeb 2.0BitDemo (music)Multiplication signType theoryDiagram
02:11
Java appletScripting languageBendingPolygonState of matterLocal ringLevel (video gaming)Archaeological field surveyWeb 2.0BuildingPolygonDatabaseSelectivity (electronic)Web browserComputer fontComputer animation
02:58
PolygonLevel (video gaming)BuildingDatabaseSelectivity (electronic)Cartesian coordinate systemBitMathematical analysisPlanningComputer animationMeeting/Interview
03:24
Magneto-optical driveWhiteboardMathematical analysisMUDMathematical analysisSession Initiation ProtocolWeb serviceIntegrated development environmentCartesian coordinate systemSoftware testingWhiteboardAreaAuthorizationMedical imagingXMLComputer animation
04:11
LogicWebsiteMultitier architectureSpreadsheetFunction (mathematics)outputFiber bundleShape (magazine)CASE <Informatik>Uniformer RaumLevel (video gaming)AirfoilMappingAuthorizationLogicProcess (computing)Shape (magazine)1 (number)Computer fileFilm editingComputer animation
05:07
Open setStandard deviationWeb serviceThermodynamischer ProzessProcess (computing)Java appletWeb servicePlug-in (computing)Projective planeStandard deviationArchaeological field surveySoftware developerWeb 2.0CausalityThermodynamischer ProzessServer (computing)Computer animation
05:57
Process (computing)Java appletServer (computing)Server (computing)DatabaseWeb servicePersonal identification numberLatent heatWeb 2.0Process (computing)SoftwareModal logicTraffic reportingXMLComputer animation
06:54
LogicRemote Access ServiceProcess (computing)Thermodynamischer ProzessElectronic data processingGeometryPrime idealString (computer science)Digital filterSelf-organizationCompilerDatabaseProbability density functionSpreadsheetInflection pointFunction (mathematics)Client (computing)Venn diagramWebsiteType theoryShape (magazine)ResultantCondition numberSoftware testingLibrary (computing)1 (number)Information securityExecution unitFunction (mathematics)Row (database)2 (number)Table (information)Maxima and minimaPoint (geometry)MappingProcess (computing)Server (computing)Rule of inferenceThermodynamischer ProzessGeometryDatabaseoutputVector spaceLie groupQuicksortVideo gameRight angleSelf-organizationParameter (computer programming)PlanningFeature structureTemplate (C++)BitSpacetimeComputer fileNetwork topologyLevel (video gaming)Web serviceDrop (liquid)Personal identification numberElectronic mailing listPrisoner's dilemmaDescriptive statisticsInternet service providerCompilation albumSoftware developerDigitizingDifferent (Kate Ryan album)CuboidCodeStapeldateiGraphical user interfaceField (computer science)System callProbability density functionObject-oriented programmingGreatest elementElectric generatorDirectory serviceWeb 2.0Matching (graph theory)Goodness of fitRemote procedure callJava appletLogicXMLComputer animation
13:17
Probability density functionSpreadsheetFunction (mathematics)Client (computing)DatabaseThermodynamischer ProzessProcess (computing)CodeVirtual machinePhysical systemServer (computing)SpreadsheetInflection pointTraffic reportingClient (computing)Patch (Unix)Computer animation
13:57
Process (computing)Template (C++)Demo (music)Online helpGamma functionPhysical lawLipschitz-StetigkeitWebsiteFisher informationExplosionQuicksortWindow functionPoint (geometry)CodeProjective planeServer (computing)MappingWritingWeb pageTemplate (C++)Software testingDifferent (Kate Ryan album)Product (business)CuboidResultantTraffic reportingComputer configurationVideoconferencingShape (magazine)Complex (psychology)DatabaseLocal ringRule of inferenceProcedural programmingGraphical user interfacePersonal identification numberPhysical systemLevel (video gaming)Proxy serverType theoryTable (information)
17:03
Data typeProbability density functionWeb serviceWhiteboardPartial differential equationStaff (military)SoftwareMappingSpreadsheetTraffic reportingInformationVariety (linguistics)Cuboid7 (number)Window functionPhysical systemScripting languageLevel (video gaming)Point (geometry)TouchscreenUltraviolet photoelectron spectroscopyComputer animation
18:12
Data typeProbability density functionArc (geometry)WhiteboardElectric generatorStatisticsTemplate (C++)Electric generatorNatural numberMappingLecture/ConferenceComputer animation
18:35
Archaeological field surveyMagneto-optical driveCASE <Informatik>NumberCartesian coordinate systemPlanningArchaeological field surveyLevel (video gaming)Projective planeTwitterMappingComputer animationLecture/Conference
19:04
PlanningMagneto-optical driveMereologyUnitäre GruppeMultitier architectureDependent and independent variablesLevel (video gaming)AreaComputer animationLecture/Conference
19:29
PlanningMagneto-optical driveAuthorizationLevel (video gaming)1 (number)Boundary value problemPhysical systemMereologySpacetimeComputer animation
20:22
Formal grammarElectronic visual displayView (database)Data typeGraphic designData miningInterior (topology)Menu (computing)DreizehnConvex hullCalculusSpreadsheetCASE <Informatik>Physical systemCoordinate systemComputer animationSource code
20:57
LogicComplex (psychology)Server (computing)Point (geometry)Web applicationServer (computing)Web serviceLevel (video gaming)PlastikkarteFront and back endsCartesian coordinate systemWeb 2.0Complex (psychology)LogicComa BerenicesXMLComputer animation
21:42
Process (computing)ResultantMultiplication signKettenkomplexPoint (geometry)CASE <Informatik>DatabasePoint cloudSpreadsheetPersonal identification numberLevel (video gaming)Computing platformSemiconductor memoryPresentation of a groupCondition numberState of matterServer (computing)Computer fileLogicThread (computing)GeometryVirtual machinePairwise comparisonTask (computing)Right angleComputer configurationWorkstation <Musikinstrument>Web serviceWordProbability density functionJava appletCodeStandard deviationPower (physics)Total S.A.Function (mathematics)Default (computer science)Instance (computer science)Lecture/Conference
Transcript: English(auto-generated)
00:08
Okay, this is the talk that's about my day job. So I have my grown-up adult email address there. The Twitter account, anybody who follows me will understand that that's not a grown-up work account.
00:22
To save you all taking pictures of this, if you go to ianterton.com, you can find the slides under Talks. They're not on the company website yet because I don't know if we have a policy for putting web talks on our website or not. Plus the fact I only finished the talk yesterday.
00:40
So I'm gonna talk a bit about using OGC, web processing services, to move business logic to the server. Or, making your web GIS seem smart, or clever. So, you know, web GIS, it's what we do. This is our key product.
01:01
It's basically a web map, sat in front of web map server. And the councils can add their data to it. I don't know what this is. Something about bird strikes outside Gatwick. That's worrying. There we go. Anyway, we can produce very pretty maps.
01:23
Sort of base data on there. There's this area of where the bird strikes are. I don't think I'll be ever flying out of Gatwick ever again, having seen that map. Suppose I wanna do something a bit more complicated than just being able to click on the map and get some information back. Which is pretty much all we can do with a web map.
01:41
And, you know, I wanna draw a polygon. Well, that's fine. JavaScript will let me do that. At one of the Open Layers demos, lets you draw polygons. It also lets you draw polygons like the one on the left, which is not actually a very good polygon. It manages to self-intersect three times.
02:01
But, Open Layers is quite happy to let me draw that, and it will let me try and drop that into my database, and it will then fail quietly, usually. And everybody comes complaining that they can't put any data into the database. Or, you can see this attempt I've had at digitizing Illinois. I got bored very quickly before following the edges
02:22
of the state, and I can assure you that if I got bored doing it, local government employees would have got bored much, much more quickly. So, suppose you want your polygons to follow the base map. I could download all of the local government, all of the building outlines for the city,
02:43
plus the roads, plus the other polygons, to their web client, and about three weeks later, they'd just about get that map rendered, and they'd be able to select things on it. But I want them to be able to do it for a WMS base map, partly because I promised the Ordnance Survey that I wouldn't let random people download
03:01
their high-quality base map data. I'd keep it safe in our database and only let them look at pictures of it. But they want to be able to click once in the middle of this building and have the whole building selected. It's apparently, it's important for making planning applications and things. And then they want to add on bits of the building surrounding it.
03:25
Or, maybe I want to do some analysis. One of our customers is called NSIPs, which is the National Strategic Infrastructure Planning Service, I believe. And they have to, when they come, somebody wants to make some natural infrastructure,
03:40
so a new large wind farm, or a new road, big motorway, say, they have to notify all of the people who might be affected by that application. So all of the councils, all of the police areas, all of the fire authorities, the internal drainage boards, the MD, that's the Army, the environment agency, the harbor authorities, there are about 30 people
04:02
that are statutorily required to notify. But then you actually want to notify the councils that the road goes through, or in some cases, neighbors of the ones that the road goes through. And we have two levels of authority, so I have to work out all the small councils
04:23
that it goes through, and their neighbors, and then all the big councils that it goes through and their neighbors. And you can tell, it's legislation, which is why they specify the neighbors and then what they must be neighbors to, in one case, and then the other one, what it intersects, and then the neighbors.
04:41
So the logic gets quite complicated, but it's legislation, you have to get it right. Oh yeah, and they're sending me a zip shape file, and they'd like back a zip file full of PDF maps and an Excel spreadsheet, please.
05:03
Somehow JavaScript's not gonna cut this any longer. JavaScript have said, I don't think we can do that. Pass it over to the server guys. So, in comes my web processing service.
05:20
Now, you've already seen three different people tell you this, it's an open geospatial consortium standard, it was developed in 2006, apparently. And it's implemented by, amongst others, GeoServer, PyWPS, and the Zoo project. Cause I took the big portion of looking at who else was in this session before I wrote my talk.
05:42
Now, I'd used, as a GeoServer developer, I obviously used the GeoServer plugin. But I could have used any of them. So, I can write a process in Java, or if I really wanted to, in Python, but I choose not to, even though we're a Python shop.
06:01
And I can run it on my server, which means I can keep the post database credentials hidden on my server without having to give them to the users. Not that I don't trust my users, but don't try and let them have access to the database if they can avoid it. And I can have my web GIS access it, asynchronously if necessary,
06:21
and the original pin specification insisted that we could do it asynchronously because their existing software, which ran on a desktop, took five and a half hours to produce these reports. So they were very keen that we could do asynchronous processes. Turns out this isn't quite as important as we thought, but we could do asynchronously,
06:41
and it's got a well-known API. It's well-documented. I don't have to write the documentation for it. I can just pass the spec over to the JavaScript boys, and they can read it. It's much easier. So, cool. So, processing WPS is in GeoServer. It uses all the GeoTools and JTS libraries
07:03
to implement your business logic. It provides a drop-in way to allow remote access to that process, and it's brilliant. It's self-documenting. So I don't have to write any documentation for that either. It's better and better. So basically,
07:20
anything I need to do, I can use my existing libraries for. I just write some annotations around the top of my process. So at the top, I give it a title. So pick and union. Little bit of a description. So return the union of the features that fulfill the provided fields that are in the geometry passed in.
07:42
Result, the union of the selected features in geometry. The type of thing you're gonna get back, it's a geometry of some sort. I don't know what you sent me. It might be lines. It might be points. I'm just gonna union it together. I don't care. It's a geometry you're getting back. And then here's my actual function call process.
08:01
So it returns geometry. It's called get features. And then it takes these parameters. So it takes a simple feature collection, a filter that can be either CQL or OGC filter, an optional geometry to union it to. So if they've already selected something,
08:20
you send me that geometry, and I will add it to whatever batches that filter. So you can have, this was when they started adding more things to the process. There's an optional parameter that's called subtract because they said, well, suppose we click on the wrong thing and we want to delete it. Oh, we'll click on it again and set subtract and I'll take it off again.
08:40
It'll be fine. And finally, an organization because there's some of our customers don't have permission to see the whole of the database. So if they specify what their organization is, then I can check against the second database table to see if they're allowed to see those features.
09:03
And I get to specify what the minimum requirement is. So the ones with a minimum of one are required. The ones with a minimum of zero are optional. And that's all the difference. That's the magic bit that wraps around my Java code
09:21
that makes it into a process. So I was quite pleased with that. So I compile it into a jar. So I just run maven compile, maven jar. Drop it into a geoserver web-inf slash lib directory.
09:42
Restart geoserver. And as if by magic, it appears in the list of processes that are available in the web processing service. And all of the arguments are documented because I've put that documentation into my annotations. Now to be honest, it takes a little bit more than that
10:00
because you have to debug it for a few days first before it actually works. But that's the principle. I could ship it to customers like that and they wouldn't know. So geoserver comes with a GUI. This is our big selling point. We have a GUI. We have a WPS request builder.
10:20
And you can see at the top here, I can select my process, pick a union, get features. It then lists the inputs I need to provide. So I can provide a vector layer or I can provide a JSON set as text.
10:46
I can provide GML as text. I can provide a WFS remote process. I can specify in here what my filter is. And you see all of my annotations are coming out at the top. That's my previous geometry.
11:05
So it can be the text as GML or it can be GJson. Fill in my organization. I can specify what I'd like my output to do. So I can do any of the outputs that geoserver provides without having to actually code any of that.
11:21
I just say return this feature collection and geoserver takes care of processing it for me. So again, I can return it as geojson. I can return it as GML. I can return it as shape files, I think even. Could even turn the shape file back. Oops. Too far.
11:44
There's also your note at the bottom of the WPS request builder. You can either execute the process directly or you can do generate XML from process inputs.
12:02
So that's what I did. Click on that and as if by magic, I get a template file. I can hand over to my JavaScript developer colleagues and I've got a note in there that says, put the filter here. Put your existing geojson in this bit
12:20
and then post it to this endpoint that I've given you. It turns out they can do that. That's easy for them. They didn't even have to read the WPS spec. All they had to do was look up how to do an asynchronous as call to thingamajiggy and I don't know. They claimed it was hard but I don't think it was.
12:44
For the pins process, the planning system, it's a bit more tricky. So basically I had to unpack a shape file, convert it into a feature stroke geometry, connect to the Postgres database, create a filter for each of these 30 different rules. I think when you got down to it,
13:00
there were about 45 separate rules that could possibly be executed. Extract the matching features from the layers, produce a PDF map. Well, I called right to a WMS at that point. I'm not messing around drawing my own PDF maps in Java. I've got a perfectly good PDF map generator sitting behind my server.
13:21
Add a sheet to an Excel spreadsheet. I used Apache POI to do that. So again, I didn't have to write any complex code. Finally, I zipped the whole lot up and notified the client it's ready. Now you remember the original system we were replacing. We ran on a desktop. They only had one copy
13:41
and it took about five hours to produce a report. They were sometimes on big jobs. They would set it running on Friday and hope that it was still finished by Monday and that the machine didn't get unplugged by the cleaners. So we had to do this asynchronously. Actually, I implemented it as two separate processes.
14:01
One to upload, unpack, and copy the shapefile into a PostGIS table because I don't want to be messing around with shapefiles. Even if the customer is wedded to them, I don't have to use them. And then another one that actually creates a report based on that feature using a prepared template.
14:22
So I have a long video that demonstrates this whole procedure that somebody worked through but I couldn't convince it to play in Chrome. So if anybody wants to see it later, I can show it to them. So this is what their PINS system looks like. As you'll see, it looks very similar to the system that we sell to local councils.
14:42
Except it says PINS GIS in the corner instead of iShare Maps. But never want to write new code. So these red blobs are, that's a trans-channel power cable. The A3 one is new roadworks. This is a wind farm.
15:01
This is where I live so I can see this wind farm from my house. So I was quite pleased with that. So that's the wind farm and the cable coming in. So at this point, the analyst has uploaded their shapefile and it's popped up, it's been put into the database and then displayed on the map.
15:23
They can click on that and you'll see that, actually I'm running this on the test server rather than the production server, which is why I've got two results back, both of which are called ANT underscore test. We encourage them to give real uses. But you'll see that there's an option here
15:43
to call create report. They click on that and it pops up this JavaScript window. They type in their project name, the project ID. I believe that they have rules and complicated procedures for how they fill those in
16:02
that conform to the legislation. We tend to just put our name followed by test and the date. You can then select whether your report is for England, Wales or England and Wales because obviously we have two different levels of local government between the two countries and some of these projects do cross the border.
16:22
If they don't, then you can get a slightly less complex report. If you've got a cross-border report, you end up with about 60 pages because you have to consult with the English fire brigade and the Welsh fire brigade and obviously they're different. You can also customize your report.
16:40
You can uncheck these boxes. So actually I'm not interested in parishes or I'm not interested in district councils. I don't care about canals or I don't care about the army, that sort of thing. But in general, the users just pick which country they're in and click okay. So that generates a request,
17:02
sends the execute request off to my server. I send it back a endpoint that it can poll and it sits up there saying generating report. At this point, the staff were thinking it's good, we can go and have a coffee, we can have some lunch, we can laze about, we've got five hours to kill.
17:23
About three minutes later, this will pop up a little window that says, your report's ready now, would you like to download it? Sadly, I was unable to screenshot that because it's a JavaScript popup and my screenshot software won't actually screenshot popups but it's basically just a little box
17:41
that says your report's ready, would you like to download it? This is what you get when you download it. So it's a variety of PDF maps and an Excel spreadsheet. You'll see that I haven't got all 30. Again, their previous system used to send them back 30 maps whether they needed them or not. I only send back the maps
18:01
that actually have some information on them because it seemed like a waste of paper otherwise. So this one, you've got level one, two, three, hasn't come back, four, five, six, seven's missing, eight and then we go on to things like railways, ambulances, fire, police, hospitals,
18:21
low-flying aircraft, outstanding natural beauty, docks, electricity generators. And as I said, the maps are pre-generated using a template. So it's got all of the required logo on it, title.
18:44
It's got the name of the project, its number, what this map is about. It's got the crown copyright license number so that the audience survey don't sue us for stealing their maps. It's got the red blob, it's the original planning application and then in this case, it's got all of the pink
19:03
which are parishes, which are smallest district of electoral or responsibility or community councils if you happen to be in Wales. And you'll notice that that's a portrait style map because it's a long thin one.
19:24
I also get back something like this which is the same thing but this is for our large areas so it's got our counties, East and West Sussex, Surrey and Hampshire. The green one is West Sussex, that's the county I live in
19:43
and then the brown ones are the surrounding counties and the pink one is a South Downs National Park which counts as a county but obviously crosses other county boundaries just to make your intersects more complicated. And just in there, you can't quite see it
20:01
but this blue blob, Brighton and Hove which is a unitary authority so it's not part of a county but it does fill up the rest of the space. But again, you'll notice that this one is a landscape map because the system took into account how big and wide the map was and said, oh, I'll make this one landscape instead of portrait.
20:24
This is a spreadsheet. I've actually got no idea whether this opens in Excel or not because I don't have Excel but it opens nicely in LibreOffice Calc and the customer didn't come back to me to complain so I'm assuming that it worked in Excel. And then there's lots of these, there's 30 little tabs down here
20:41
and they list what are the hospitals in this case, tells you which data set it came from, what the name of it is and it's got the XY coordinates. They don't actually need the XY coordinates but the old system gave them to them so they said they should get them in the new one. So in conclusion, looking for an easy way
21:02
to make your web map application smart without tying yourself down to a backend server and to be honest, if GeoServer exploded tomorrow, we can no longer use it. I could recreate this in PyWPS or in Zoo in a few days. There's nothing special about the fact I used GeoServer.
21:23
I would recommend you think about using WPS to move your complex business logic into the server. And as a reminder, the talk can be downloaded from ianterton.com slash talks.
21:47
I disagree with you because I think that having WPS within GeoServer is something very special because you can use all the power of GeoServer. Yeah, totally. So any other questions or comments?
22:01
Oh yeah, too many. I saw the first hand over there but. Thanks for the talk. So how customizable is running the WPS on GeoServer in comparison to PyWPS for example? So I'm quite new to WPS in general but now you had quite a simple task,
22:22
geoprocessingly speaking, but how customizable is? How do you mean customizable? I can run any process I like. So anything I can code in Java, I can run as a process. Right, but I mean now it was essentially an intersect with the parishes and the other layers.
22:41
Slightly more than that in that I have to do an intersect and then I have to find touching for different, and it's across three different layers. So anything, basically anything I like, as I say, and then I can put them into an Excel spreadsheet or I can put them into a WMS called PDF. So anything you can do with GeoTools or JTS
23:01
or that you can write your own Java function for, you could do it. Okay, or Python. You can in fact use Python inside GeoServer as well, yes. That is an option. But it's not a very efficient one. Who's here? Here, yeah, sorry.
23:21
Hello, thank you for the presentation. The previous session dealt with WPS on QGIS server and it evolved a lot of dealing with infrastructure, the platform and not the business logic. So from those two presentation, I may conclude that GeoServer in its current condition
23:43
is as a platform suited much better than QGIS server with PyWPS because actually what you demonstrated, you didn't deal with fork in the platform. You just put your business logic code into ended plate.
24:01
Can you comment on this? Yes and no. In that GeoServer takes care of a lot of the complexities for me and it will create me a new thread for each process. So it's a new Java thread. It's not giving you complete isolation the way
24:21
that say a Docker container would do in PyWPS. So you could theoretically write a process that clobbered itself if somebody ran it twice. It's harder to do because GeoServer takes care of creating all the files for you and things in the background and it knows what else is happening. So you've not got absolute isolation but you've got thread isolation.
24:41
So provided your machine is big enough to run several of your jobs at the same time, you've not got a problem. And obviously we're running this in the cloud so if there is a problem, we just put it on a bigger instance. But as I say, they don't actually run this job very often and the pins one which is the big one. The picking features out of the map
25:01
is a tiny little process. It hardly takes any memory at all. So most of the work goes on in the Postgres database and that's fine. So even when there's hundreds of people doing things at the same time, it doesn't really notice. Okay, thank you. Very similar to, as you said, PyWPS default setup
25:23
and it seems to be working for most of the use cases. Yes. Is there any concept of state in the WS standard as in you can only run this process at one, you can't run, similar to what we were just saying, you can only run one process at a time or something. No, they're completely independent processes.
25:43
And you can either run them and wait for the result which is what the toy picker does or you can run them asynchronously which is what the pin service does and then you just get an endpoint, you poll until it says I'm ready, here's your results. This is the URL your result is at. So you can't really stop multiple processes
26:01
running at the same time? No, the whole point is to allow multiple processes to run at the same time. I actively encourage customers to do that. Exactly. More? Doesn't seem to be the case, so thank you for more time.