Deploying A Open Source Web-GIS Within Warwickshire County Council
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 95 | |
Author | ||
License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
Identifiers | 10.5446/15523 (DOI) | |
Publisher | ||
Release Date | ||
Language | ||
Production Place | Nottingham |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
|
FOSS4G Nottingham 201323 / 95
17
25
29
31
32
34
48
50
56
58
68
69
70
82
89
91
00:00
Level (video gaming)Well-formed formulaLogarithmNatural numberPixelInterpolationZoom lensRaster graphicsSequelValue-added networkVector graphicsGroup actionScale (map)ConsistencyLocal ringComputer configurationDew pointPairwise comparisonVector spacePiArmRaw image formatOpen setAxonometric projectionServer (computing)Execution unitProxy serverWeb pageGeometryReverse engineeringMaxima and minimaNumberCache (computing)Dependent and independent variablesEmailDensity of statesLocal GroupConditional-access moduleRadarMetropolitan area networkCompilerGoogolAlgebraic closureScripting languageError messageData compressionInformation managementWide area networkSpecial unitary groupMoving averageComputer clusterTwin primeLine (geometry)Gamma functionBefehlsprozessorFeedbackService (economics)Bit rateSupremumBand matrixNormed vector spaceTotal S.A.KnotPerformance appraisalLevel (video gaming)TesselationRaster graphicsVector spaceMaxima and minimaEmailPerformance appraisalGroup actionScaling (geometry)Video gameRegular graphProjective planeStrategy gameService (economics)Server (computing)FeedbackGeometryCache (computing)1 (number)Algebraic closureCartesian coordinate systemCompilerSearch engine (computing)Dot productZoom lensWeb browserBookmark (World Wide Web)Information securityMiniDiscSoftware testingOpen setRight anglePixelBefehlsprozessorParameter (computer programming)Dependent and independent variablesMappingCASE <Informatik>Ideal (ethics)NumberAuthorization2 (number)Proxy serverPhysical systemComputer configurationImage warpingWave packetCore dumpVideoconferencingLine (geometry)Video projectorFlow separationRule of inferenceStructural loadProduct (business)Default (computer science)Local ringDatabase normalizationDenial-of-service attackFehlererkennungSI-EinheitenSpacetimeArithmetic meanAreaBitImage resolutionQuicksortLattice (order)LengthWeightMultiplication signStaff (military)Computer filePlastikkarteState of matterInheritance (object-oriented programming)Form (programming)MathematicsExecution unitOrder (biology)Different (Kate Ryan album)MassTable (information)Constraint (mathematics)InterpolationPoint (geometry)Computer animation
09:43
Open setSoftwareLetterpress printingLevel (video gaming)Single-precision floating-point formatEmailSoftware bugRevision controlElectric generatorMereologyOpen setHypermediaResultantSet (mathematics)PlanningPhysical systemService (economics)Open sourceTime zoneArchaeological field surveyQuicksortSingle-precision floating-point formatCartesian coordinate systemAuthorizationOffice suiteSampling (statistics)Solid geometryCASE <Informatik>NumberDatabasePoint (geometry)Group actionRandomization1 (number)Web 2.0Boss CorporationAreaBeat (acoustics)SoftwareStaff (military)Translation (relic)Internet service providerMoment (mathematics)AverageSoftware maintenanceCanadian Mathematical SocietyUser interfaceQuantumDemosceneRight angleMathematical analysisMultiplication signMobile Web2 (number)BitBootingComputer hardwareOrder (biology)Event horizonVideo gameIdentity managementPairwise comparisonRow (database)Population densityPhysical lawFood energySoftware developerWhiteboardWebsiteMassSpacetimeSerial portProper mapInsertion lossSystem callPerpetual motionObject (grammar)Texture mappingProcess (computing)Content (media)Control flowScheduling (computing)Special unitary groupNetwork topologyComputer animation
19:26
Computer animation
Transcript: English(auto-generated)
00:00
And then you enter into your favorite search engine. You don't get a love heart like last night, but instead you get the answer of six in this case. So we know to use six levels of tiles is sufficient. If you create more levels of tiles, well, it's kind of redundant. So there's no need.
00:20
You just use more disk space for something that's unnecessary. The disadvantage of this method is that when you zoom at a level other than the one that the raster resolution is, you end up with something ugly like that, because it's interpolating on the fly.
00:42
GeoServer's interpolating on the fly. We want it to look a bit better than that, because these are tiles we know that they're always going to be shown at a fixed scale. So there is a better way. We pre-interpolate. So basically, we create a raster
01:01
for each of the zoom levels that we have. So as you can see, there's actually about 11 zoom levels for the rasters. And each one we create a separate raster layer for. And that raster layer will always look very good when you zoom to it at that level. And the way you do that, you're basically just resizing
01:22
your original raster ahead of time, which is what that top line is doing there. So we resize it. I'm using the Lanczos interpolation, which from various testing gave us the best quality. But it's really slow. It doesn't matter, because we're doing it ahead of time. You don't want to do it on the fly, though.
01:42
I don't think it's even an option for on the fly. As I mentioned, you've got one raster is one zoom level. So you end up with 10 rasters, which is fine. It's a little more disk space, but not actually that much. And the parameter that you need is the TR in GDL warp
02:04
and that resizes it. The pixel size to use is from the pixel size column in the grid set on geo server. I don't remember how that works, but it worked. I did it several months ago. I can't release things.
02:21
Yep. There is a third option for your base map tiles and that's to use vectors to vector data like master map or open street map. We don't have open street map because it's a pain, but we do have master map loaded. So you can use vectors to create a base layer like with ideally using geo servers layer groups
02:44
if you're using geo server. So we have about six different layer groups. So we've got one layer group for each product and then we've got a layer group of the five layer groups and that in turn creates an entire zoom stack
03:01
so to speak of all of those products. And this is a vector map to one. You should pre-seed it because otherwise geo server tries to render it on the fly, which is okay when you get really zoomed in, but when you're zoomed out, it's trying to load millions of features from a vector map local or vector map district
03:22
or even meridians actually got quite a lot of feet, probably a couple of million features in there. But it's a lot of manual effort to create those layers because you've got to style up. In this case, there's 53 tables, but there's a lot more layers than that within those tables. There's probably one, 200 different styling rules.
03:40
The outputs, I'm not sure how much of that's going to come through on the projector. That's the original, the first method. The second, the one where we've pre-created our tiles, the one raster for one zoom level is in the middle. As you can see, it looks a lot better
04:01
at the right zoom level. And the third is the vector and they all have different advantages and disadvantages. This one's ideal for if you're still using it as a WMS because you don't gain the advantage of having them one-to-one. The rasters are good if you actually want a raster base map,
04:22
because some people like their OS mapping. And of course, vector data, we can style it however we want. So what we have is a nice contiguous styling from strategy all the way down to master map. And it all blends in as compared to regular OS rasters, which generally look a mess.
04:45
Let's see. Yep, projections continue to be a pain, or at least they're a pain for me. I don't know about everyone else. So what we basically do is have everything in British National Grid, because that's easiest.
05:00
The other thing is to make life a lot easier. The projection scales that you use, specifically for tiles, this one is, projection scales you use in the OpenLayers application should be the same as the ones on the geo server. If they're not, your tiles will end up in somewhere else.
05:21
So they get slightly misprojected, or at least they always do for me. So we also worked in conjunction with the OpenStreetMap GB project, and they actually converted their tiles to be the same as ours. So everyone else kind of has to use our tiles now for that project.
05:41
There are a couple of gotchas with OpenLayers. You do need a proxy if you're gonna use the WFS, because otherwise what will happen is you'll end up, your browser won't allow you to make the request otherwise because of security concerns. The bigger hidden gotcha is you need to declare your DPI
06:02
in OpenLayers because the OpenLayers uses 72 dots per inch as its default, whereas the geo server's dots per inch is 90.714 something, for I don't know why. So if you have the wrong dots per inch, again your tiles are going to end up in the wrong place.
06:21
That one took a while to figure out as well. Geo server has a, well, a couple of gotchas, tips really, lower your WFS maximum number of features. The default's a hundred thousand, meaning if someone makes a query, they can get up to a hundred thousand features returned. That'll kill any browser and probably your server. Seven and a half thousand's what you set it to
06:41
for our internal systems. For the external systems, we're probably gonna set it to 750. That also means that it's harder for someone to perform a denial of service kind of attack on it. The other thing is if you set the, you need to set the response cache headers for each individual layer, and you need to do,
07:00
if you want that layer to be, the tiles to be cached on the client side. So you don't have to keep serving out the same tiles again and again to the same users because they're all generally looking at the same sorts of areas. Going live. Smaller is better. We used the Google bit,
07:21
the Google closure compiler to basically turn all of that into one minified file, minified JavaScript basically being a bunch of gibberish to humans, but it works. It's not much smaller as you can see, but it's faster to load and it kept with the bonus
07:40
that the Google closure compiler actually has a error checking in which alerts us when someone's done something wrong, which they have this week, but I'm not around to fix it. So, yeah. We went live cunningly on a Friday afternoon. The idea was to spread the load. So half the council would do it on a Friday afternoon. The other half would log in on Monday morning.
08:02
As you can see, I sent the email there like a few seconds later, half the authorities on it because they've got nothing else to do on a Friday afternoon. Yeah, this is, I think, eight cores of CPU. So you can see they kind of hammered it, but it did stay up and, oh, I went too far there.
08:24
Ooh, I'm there. And there was no lag during it because, again, it's a tile service. It was fine. Yep, we got lots of them. We served about 70,000 tiles that day, peak requesting at 41 tiles a second.
08:41
The minimum since then is about 12 tiles a second, 15,000 a day. As a bonus, we have lots of OGC services we can now serve internally. They can be used by any of our systems which we're planning to use. Ideally, we'll have them publicly available, but that's kind of wishful thinking
09:02
because security have to actually say yes. They're gonna get to it eventually. Evaluation and conclusions, she's scowling at me. We've had excellent user feedback so far. Very impressive, looks excellent, better than the old system, that's not saying much. Yep, we also created a video as well
09:22
saying how to use it because some users said we want something, how do we use it? Turns out no one ever actually watched it. One person's watched that video from start to end. So, you know, if users say they want training materials, they may be lying. At least they all asked where.
09:40
Bugs, there are actually, there are bugs, yes, in systems. These are sort of just the ones I reported. I report a lot of bugs, I get a lot of scales, especially from proprietary systems because despite the fact that those numbers do look quite bad, it's still better than proprietary software. You report bugs with Esri, goes into the system, they deny it's a bug. Eventually, if they accept it's a bug, it goes to America,
10:02
you never hear from it again. In this case, with open source, we can pay someone to fix the bug and it gets fixed. Yep, costs, as I mentioned, it's a lot cheaper. There isn't a purchase cost. We decided to put some extra money into development, which is where the 10, 30,000 comes from because we're gonna spend a little more.
10:22
Maintenance is cheaper and it took a lot less staff time to do. So the total cost was cheaper. Conclusions, as you might notice, I'm sort of wrapping up now very quickly. There are lots of pros. It's easily implemented and extended. We're not reliant on a single vendor anymore. Previously, as mentioned, we had something not working for two years
10:42
where our vendor couldn't do anything about it because they're incompetent. This is being recorded, right? I haven't named them, right? Okay. So we've got a single vendor, they can't do it, but in this case, we have lots of vendors. It's a JavaScript client, okay?
11:00
You know, half the people in this room can probably fix bugs for it if we throw money at them. So that's a great big advantage. No more of those are licenses. It's a very big one because I really hate licenses because they just keep getting in the way and causing lots of grief. Again, it's much cheaper. The cons are there are, yes, still bugs, even with the open source stuff,
11:21
and it does require an internal skill set. If you go externally, which you can do, because there are lots of vendors here, they'll charge you lots of money. It will still possibly be cheaper, and you still get a lot of the other benefits, but if you're doing it for savings, there you are, and a minute or two late. So that's what it looks like, the advanced version.
11:42
Do we have any questions? Thank you. Thanks for this detailed work, and very impressive. So now we have about 10 minutes for the question session for all of our three speakers here.
12:01
Do you have any questions? Yeah, what sort of searches and gazetteers do you have flowing through the web interface there? Pretty much anything are database searches. Because the searches are all WFS, it's basically querying anything in the web feature service that our gazetteer has.
12:21
So we've got the NLPG, the National Street gazetteer, 150,000 gazetteers, a couple of internal searches, road numbers, there's a whole collection of them. That's using postcode searches, I think the one on the right is a postcode search.
12:44
It depends how good our database is feeling that day. You can get a result inside of a second, sometimes it's a bit slower, not sure why. We've indexed the columns, but it's usually within a few seconds. Yeah, yes.
13:05
Oracle, because all of the council, the council basically has an Oracle database, that's what we're using, corporate database. I'd like to use PostGIS, but there'd be no advantage because the rest of the authority is still mostly on Oracle, so there won't be any saving there for the authority because we're still paying them
13:21
five, six figures for nothing. Oh, lots. Catch me afterward if you like, but yes. I just came in a bit late, and I'm just saying a lot more much. Yep. But I can see that what you've replaced is virtually everything that your internet service is,
13:45
and it's a very, very new, I've looked at Heron a bit, and I've also looked at the kind of managed print, but I was just thinking about providing one service unit, kind of like the planning or something like that, to sort of comprehensively remember the...
14:02
Yeah, yeah. We're trying to replace all of our previous systems with this one. So far, only about half the users have transitioned. Being account counsel, as ever, there are people who don't like to progress too much for one reason or another,
14:21
so they're still using the old systems. We're going to push them over, whether they like it or not. We're still running the old system. We just have no support, so if it breaks, it breaks. Ideally, it will break someday soon, so I can get rid of it, but it hasn't yet. I haven't broken it intentionally yet. That might be tomorrow. Yes, sorry.
14:41
Is there, or are there any plans to integrate with other counsel systems, such as planning applications to be able to use? Yes, yes. Basically, we use FME behind the scenes to get all of our data into our corporate database. FME's data translation loader is a proprietary one. They're actually very good at fixing bugs. They're one of the rare exceptions,
15:02
but we've, so we just need to be able to read their data and stick it into our data store, and the second it's in our data store, we can access it through this. Conversely, our systems can access this using the WMS, the WFS, any of the OGC services, but again, proprietary systems don't usually do OGC services,
15:21
or if they do, they don't do it well. Even ArcGIS has quite a lot of bugs with them, but then QGIS does too. Sorry, I have a question back there, or maybe you left. Yeah.
15:40
Survey tool in one of your demonstrations, and I wonder what kind of hardware is being implemented now, like a data portal, like mobile phones, or it's a computer? At the moment, it's a survey design tool. So, it's for random stratified surveys.
16:03
You can create your strata from a basic coastline in QGIS, then create your strata into zones of high low media abundance, whatever, and then generate random transits or points to sample within that strata. We have three surveys going here at the moment
16:21
designed by that, and the next step will be to work with the people analyzing the survey results to add an analysis tool to the survey design tool so they can then design the survey, carry out the survey, and analyze the results as part of a single tool.
16:43
How does the SilverStripe CMS that you had developed, presumably that prior to it? That's open source? It's totally open source. It's PHP-based, it's PHP-based. Its claim to fame was for the democratic,
17:02
the last democratic election, they used SilverStripe for their website, and as you can imagine on election night, it took quite a few hits, and it coped, obviously. So it's not the 800 pound gorilla like WordPress or Drupal,
17:23
but it has its own solid user base. The chief technology officer is an XGIS person, and the opportunity to work on a tool like this within a CMS made his heart beat with joy. His boss said the weekend after we interviewed them for this job,
17:41
I've never seen a German so excited. I have another question for our second speaker, just personal interest. I wonder, for the majority companies and individuals in Korea, what's the most popular software,
18:02
just for mapping? I know in most areas it will be either ArcGIS or Quantum GIS. I just wonder if that's the same case in Korea at the moment. Actually, around 50% of Korean market is dominated by SC product, like average GIS and others, but with the advent of the QGIS nowadays,
18:25
now we can see the growing number of QGIS users. Just a quick comment, because I'm from China, and I think in China at the moment, most people are still using SRI products,
18:40
and at least when I was in university, that was six years ago, and I was taught ArcMap 9.2, I think. So yeah, I think open source software is quite widely accepted in China.
19:03
That's just my personal point. So do we have any more questions on this session? I think we're a bit late, so the next session starts at two quarter past one, so you have about seven minutes.
19:21
And if you don't hear it, Greg, yes, you're good, or maybe you're much better. Thank you very much for your time. Thank you. Cheers.