MapServer, Optimize for performance
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43441 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
MappingSet (mathematics)Computer animation
00:21
Revision controlLevel (video gaming)Instance (computer science)Artistic renderingComputer fileCodeScripting languageDigital photographyExpressionArchaeological field surveyExpert systemCASE <Informatik>Revision controlPhysical systemMappingResultantSocial classMathematical optimizationNumberSampling (statistics)Function (mathematics)Software developerRow (database)Bit rateStructural loadPreprocessorProcess (computing)Total S.A.Frame problemMultiplication signHypothesisProjective planeGraph coloringWater vaporCollaborationismService (economics)InformationServer (computing)Cone penetration testFigurate numberStatement (computer science)DigitizingDifferent (Kate Ryan album)Network topologyShape (magazine)Reading (process)Set (mathematics)Proper mapComplex (psychology)TesselationSubject indexingMultiplicationVertex (graph theory)Right angleElectronic visual displayStack (abstract data type)Exterior algebraRegulärer Ausdruck <Textverarbeitung>Grass (card game)2 (number)Raster graphicsLecture/Conference
10:12
CoprocessorAuditory maskingServer (computing)State of matterCoefficient of determinationProcess (computing)Table (information)Subject indexingRoundness (object)1 (number)Instance (computer science)Mathematical optimizationPolygonScripting languageExpressionTesselationVolumenvisualisierungMultiplication signLevel (video gaming)CASE <Informatik>StatisticsQueue (abstract data type)Type theoryComputer iconCalculationFigurate numberExtension (kinesiology)Complex (psychology)Functional (mathematics)Standard deviationComputer fileLine (geometry)Artistic renderingRevision controlSymbol tableLabour Party (Malta)CoalitionGrass (card game)Network topologyProjective planeField (computer science)Shape (magazine)WebsiteOpen setExpert systemVariable (mathematics)Latin squareGraph coloringMeasurementParallel portGreen's functionCodeSet (mathematics)AreaMereologyCuboidMedical imagingNumberRow (database)Sign (mathematics)PreprocessorCollisionRaster graphicsRegulärer Ausdruck <Textverarbeitung>AuthorizationCache (computing)Electronic visual displayWindowProper mapCore dump2 (number)Vector spaceDifferent (Kate Ryan album)CalculusLecture/Conference
Transcript: English(auto-generated)
00:07
OK. We'll get started for our second talk of this session on MapSolver. So our next speaker is going to be Lars Schilberg from SAB in Sweden. Lars is going to share with us a set of tips and tricks for optimizing your maps with MapSolver.
00:26
Thank you. Originally, it had been Jeff and Mike that had been doing these tips and tricks. So I do it a little differently today. I will just have two use cases, and I will work them
00:45
through a little more. I will talk about complex road shield labeling and how to optimize this use case. And of course, it involves a lot of g.org stuff. So be aware, this will be quite technical.
01:02
And finally, I will talk about tight tile indexes for raster maps. And both these things are things that I don't know if they're really new, but it's things that I came up with during last winter.
01:22
So my background is that I'm a land surveyor. I'm PhD in cartography. I used to be a grass developer and a user in late 80s and beginning of 90s. I started to use MapServer for the first time in 2001, 2002 frame.
01:42
I used it then for a big spare time project. Then I was away from MapServer. I worked as a company specialist for digital maps in my company. That is SOB, the airplane manufacturer SOB. And what I've done in recent years is this MacM project
02:06
on GitHub. I don't know how many of you have seen that project. Yeah, not very many. Actually, it's C-shirts with MapServer.
02:21
And how to configure MapServer to read S57 files and render them really fast. And I really love to do beautiful WMS services.
02:40
So I love map rendering. And for instance, this is an example here from Smachem. It's a cooperation between Simon from MapCures. Well, the MapCures team where we did proper symbology for lighthouses, for instance.
03:02
We had a couple of different versions. And this is now in there with the optimized version. I won't talk about that today. I will talk about complex roadshow labeling. Many data sets contain multiple road shields
03:22
for certain segments of roads. So you see here, I think you call it code sharing also. And most online maps don't display it. And my goal was then to do this automatically
03:40
with proper stacking. And I know the MapServer has the abilities to do it. Some examples here from the National Land Survey of Sweden. You could stack it along the road. You could stack it perpendicular. Or you can stack them on top of each other.
04:02
And the example I will show you today is stacking them horizontally. No, vertically. Right. So this is what the data looks like. So you have then a road number one, an alternative road
04:22
number, and a third alternative road number in some cases. Most of the time, this could be empty. Or most of the cases, it's just one road shield. So I have small extracts of this.
04:49
But I place them on a Gist GitHub. So you could go to my account after the presentation
05:01
and look at the full examples. So the map file syntax for doing this is in my first version that I did, that I did many years ago, is that you have a class with three shields stacked. You have to have a first expression, a class expression,
05:24
and then a label expression. And then comes the background of the shield. And then comes the actual shield that's placed on top of the background.
05:44
So this is, of course, if you're talking about performance, it's always a matter of filtering how much information that goes into the rendering. So you have actually three main ways to filter data. First, you could filter data with a filter command
06:04
on the map level. And then you could have class expressions and finally label expressions. So this is then regexp.
06:21
For instance, I would like to find out the road number that equals e and one character. And that's because I want to have the right size of the shield. So what I do here is that I end up
06:41
with a lot of label expressions. It's quite massive. So if you look at this file afterwards, I think just doing these labels, it's about 800 rows of map file code.
07:01
So you have the case for two characters. And I have to tell you, most European road shields are green for European main roads and then blue or other colors for secondary roads. So in Sweden, it's blue for the secondary roads.
07:27
But they still can be main roads. And they could be up to number 500. But I should not display them if they're above 500. So we end up with a lot of expressions here.
07:45
And so as you saw before, it was ultra photo. And I rendered the roads and some other roads and then the road shields. And it takes almost half a second.
08:02
And it's a total of 900 milliseconds. And it's a little too much. By the way, you get this kind of information if you run maps with debug mode, usually three or five.
08:20
So what to do? I found out this is working. I get nice results. But the rendering takes too much time. Half of the rendering time is then spent on the road tree labeling. So my hypothesis here is that the expression statements
08:45
are very costly. So there must be some way to improve it. So if I could then determine beforehand how to just get it
09:04
down to three expressions, I could then pre-process what color it should be and how wide the shield should be. You see, the shields are wider the more text there is. So I made a script in OGR to OGR.
09:25
And this is done with shape files. But since it's g.ogr, it would be the same if it's package or post-gis. So the full example of a script for doing this pre-processing you find here,
09:43
is road science processing. So what I do here is I take the original file and output to a new shape file. And I use the SQLite syntax in my expression.
10:02
So we will look then at what I do in this SQL expression. And it's 25 or 30 rows of code. So what I found out is that if you have SQLite, you could then do the same regex in SQLite.
10:27
So I then pre-process. For instance, here I pre-process what shield size I should have. So I actually determine what the symbol should be,
10:40
if it's a narrow symbol or a wide symbol. And I pre-process the color, yes, the color. So I know if it's going to be blue or green. And finally, I also pre-process which case it is.
11:05
Is it three shields, two shields, or one shield? And I also check that it's not empty.
11:22
So I filter away all the empty ones. And then, of course, when you have done this, don't forget with your normal optimizations. You could optimize the table size. You could optimize on creating index
11:43
on the most used variable and create this spatial index again. That's kind of standard, I hope. So we end up with the row data with new variables. So it's the symbol for box one, two, three, the colors,
12:04
and the number of signs. So now I can write a new map file. So this has only three expressions. Expressions case three shields, and so on.
12:26
So it's the same then, expression with two and one. So let's run it again. So now I'm running with a new version. And I'm down to 24 milliseconds.
12:42
And the total rendering is down to a little more than half a second. So I think that's a quite good improvement. So I was very satisfied with this pre-processing for the road shields.
13:02
So 24 milliseconds compared to 400. So that's kind of nice. So it worked. It's cutting down the rendering time 15 to 20 times for that layer. And next level of complexity is to display road names
13:21
at the same time. But I didn't do that. Actually, I was displaying the road names at the same time as the roads and then for the lower levels. And that worked. Then label cache takes care of the collisions and so on.
13:43
Next one here is raster tile indexes. If you get a lot of rasters, usually you just run g.t index on them. But it's a little slow in some cases. So here I have a big file that I got over Finland.
14:04
But I only have data in certain parts. And they were quite big, the raster images as well. So I did do retiling on them. And I then remove empty areas.
14:22
And I remove parts where I don't have data. So I made a process in GDAL, of course. Here is another case, raster images over Norway and Sweden. It was transparent where I show it in black there.
14:46
So the full code example here for that is found here. Import raster retile. So I did own version of retiling with mask. Because GDAL retile is not handling no data properly.
15:06
So I rewrote it in bash. You could find it on my gist. I removed empty tiles. You just run. If you have to unset no data, you run statistics on it.
15:22
If it's standard deviation zero, then you remove the tile. And then I trimmed the tiles. And I create a mask with GDAL calc around where there's data.
15:40
And then I make a polygon around the mask. With GDAL polygonize. And then I trim away. So I then read the extents from OGR on the vector that I did around.
16:02
And then I check if it's the extent on the original file. And I can then compare them with binary calculator here. This is all bash, by the way. So OGR extent and GDAL extent are bash functions.
16:25
And so if it's the same, then I keep the file. Otherwise, I GDAL translate with a new project window. And I do a new GDAL t-index and do shape tree on it
16:47
to have a spatial indexing on it. And of course, this was about optimization.
17:03
So I was doing this on quite big data sets. And running GDAL and OGR commands, you gain a lot if you use the parallel functionality in Linux.
17:20
So I use new parallel. And then I could set up that use all the available processors. So here, for instance, I've set up plus one. So I'm using, for instance, when I was running these scripts, I had a server with 16 cores. So I set it up plus one.
17:41
So it's always one job in the queue. So I could speed up that pre-processing a lot. So what happened here is that when I didn't have the tight tile indexes, so for instance, when I was around the borders
18:00
here, very often, it had to read both the Swedish and the Norwegian data set. But when I have these tight tile indexes, that happened very more seldom. So actually, there's a speed improvement by 100%, because it doesn't have to open the other countries' rest files.
18:26
Well, if you're really at the border, it depends on how tight you do the retiling. But rendering becomes much faster, noticeable. I don't have any figures, but it was a big improvement.
18:42
And this process can be done both on palette, RGB or RGBA. It's mostly the checking for the empty tiles that is different here.
19:01
Final tip, doesn't have to do with that server. But I thought I should mention it anyway. I just found this command line, cs to cs. I don't know, how many of you are using that? Yeah, quite ten or so, yeah.
19:20
That's very handy when you author map files, because I author map files in many different projections. So if you're going to paste in the extents of something or... So I just love that. And I found out this winter, I don't know why I've missed it before.
19:42
Maybe it's because it's one of the command line tools in Proja. So that was my two use cases.