MapServer #ProTips 2
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 183 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/32139 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer | ||
| Production Year | 2015 | |
| Production Place | Seoul, South Korea |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSS4G Seoul 2015124 / 183
7
8
47
53
54
65
73
74
79
82
84
92
102
103
105
124
126
127
130
141
142
143
156
161
162
170
176
178
181
183
00:00
Goodness of fitRight angleComputer configurationProjective planeError messageLevel (video gaming)StapeldateiWindowWeb serviceAreaTexture mappingServer (computing)Configuration spaceString (computer science)Uniform resource locatorEmailClient (computing)Normal (geometry)Query languageHypermediaMultiplication signWeb 2.0Subject indexingIP addressInstallation artWordChainData loggerRevision controlOpen sourceProcess (computing)Computer fileIntegrated development environmentTrailInformation securityDebuggerCodeOperator (mathematics)Physical systemUniverse (mathematics)Crash (computing)Variety (linguistics)Power (physics)Electric generatorImage resolutionSoftware bugMathematicsType theoryCartesian coordinate systemTracing (software)Software developerZoom lensInformationTraffic reportingMappingCore dumpQuicksortDivisorDefault (computer science)Bit rateLoginData miningMassExecution unitGroup actionUtility softwareAddress spaceWebsiteElectronic mailing listMaxima and minimaRange (statistics)Software maintenanceMereologyShape (magazine)Binary codeCollatz conjectureXMLComputer animation
08:27
SequelServer (computing)DialectType theoryTexture mappingDevice driverCuboidFilter <Stochastik>Multiplication signDatabaseShape (magazine)DivisorPower (physics)Computer fileLevel (video gaming)Existential quantificationConnected spaceSeries (mathematics)OracleWeb serviceGraph coloringRight angleQuery languagePoint (geometry)Parameter (computer programming)Interface (computing)Mathematical analysisAreaKey (cryptography)Similarity (geometry)Error messageOperator (mathematics)MathematicsMassTable (information)SphereBitVector spaceProjective planeStructural loadGame controllerRule of inferenceObject (grammar)QuicksortFrequencyLibrary (computing)Order (biology)Client (computing)Web pageDefault (computer science)Communications protocolView (database)InformationComputer-generated imageryMappingExtension (kinesiology)Different (Kate Ryan album)Product (business)Physical systemMessage passingAttribute grammarStandard deviationSystem callInternet service providerCountingAutomatic differentiationAdditionFront and back endsSoftware developerComputer animation
16:49
Different (Kate Ryan album)Template (C++)Variety (linguistics)Web serviceParameter (computer programming)Type theoryServer (computing)Source codeView (database)Electronic mailing listAdditionInformationTexture mappingComputer fileFile formatRight angleFinite differenceProjective planeLine (geometry)Cache (computing)Field (computer science)Block (periodic table)Asynchronous Transfer ModeTouchscreen1 (number)VolumenvisualisierungMedical imagingCode40 (number)Inheritance (object-oriented programming)Lattice (order)QuicksortLocal ringCompilation albumPoint (geometry)Computer-generated imageryMobile appPower (physics)Goodness of fitComputer-assisted translationMassMultiplicationCompass (drafting)Software testingChannel capacityMathematicsVarianceForm (programming)ReliefLarge eddy simulationGraph coloringShared memoryOverlay-NetzGame controllerPhysical systemMereologyMappingSpacetimeFigurate numberEndliche ModelltheorieSet (mathematics)EmailTheory of relativityMetadataPlotterCuboidComputer animation
25:10
Computer animation
Transcript: English(auto-generated)
00:05
Howdy. Good afternoon. So we're gonna tag team here and do it together. I'm Jeff. That's Mike. No, I'm Mike. He's Jeff. Right. So, well, let's get some background on us and how long we've been working together.
00:23
I mean, not our team. It seems a very long time, Jeff, that we've been working together. Yeah, yeah. So how do we describe each other? We're sort of both the power users of MapServer, right? The ones remaining, yes. Well, yeah, so, so yeah, although you do write, you're a core committer.
00:43
I have written some, I'm not a core committer. I'm on the PFC, but I'm not a core committer. Yeah, I have written some of the code. Yeah, even I have, believe it or not. I've even written some of the code. So you see there's a low bar to putting code into MapServer. Right. So that's, that's probably where we're coming from, just using background.
01:02
So we're like the power, like we're not going to blow in there. But we are power users of MapServer. And we've been doing this for 14 years now. Yes. Yeah. 15 years. 14 years. Okay. Whatever. You're on me. So yeah, that's what you're getting today. You're not, you're not getting the two developers, but you're getting two,
01:21
almost, developers. Okay. So yeah, we're gonna talk about power tips and some of them are pro tips, sorry, power tips, pro tips. Some of them are maybe simple. Some of you might have already seen before.
01:40
But yeah, so we'll go through pretty quick. There's not a lot, right? Is there a lot? You should know. So the first one that I've already skipped to, I don't, I don't know how we're doing this, but it's going to be debugging. So you're building an application. You're doing it with like, you saw the, the open layers, you know, front-end stuff. And then how are we going to debug
02:05
errors? Mike. Well, you set the debug at the, at the map level or at the layer level to see which types of debug information you want. There's even the possibility of doing GDAL, OGR debugging,
02:22
embedded right in Map Server. So you can add this special config option and then see GDAL errors in the Map Server log file. And then this is your part. I'm, I'm the Shaped IMG guy. Yeah, yeah, I kind of live at the command line and I find the most powerful debugging thing is really just one command line tool Shaped IMG. And
02:47
yeah, I mean many people know of it, but I don't know if many, if so many people rely on it, you know. So yeah, you get the best of both worlds here. Yeah. But what I love about Shaped IMG is it's pretty much if you install Map Server,
03:04
the Map Server utilities will be there. So if you're in a hosted environment, likely it's there. If you built it yourself, it's definitely there. If you're working on a paid client's machine, it's going to be there. So that's why I turn to that. And then yeah, there's some important switches.
03:22
We listed all underscore debug. It sort of, it sort of lists the maximum amount of debugging info. Yeah, right? That's probably not what you want to start with. Right. But sometimes when you need all the debugging information, you can use the all debug switch. Right. So,
03:40
right. Right. You? So another method you can use is avoiding even running through the web server at all. You can call Map Serve with the dash, dash nh, the no headers option. And then you just pass it a query string, which would be at your actual URL. And you, it's the same as executing Map Server through Apache,
04:02
but you don't have to deal with any of the issues that you might be having with Apache. So you can isolate your issue to maybe something with Map Server versus something that's occurring directly because you're running it in a web server environment. Maybe it's a permissions issue or other things like that. So it's good to isolate where your problems might be occurring. And this is a good way of doing it.
04:22
Yeah. Yeah, I just want to reiterate on how important that is, Map Serve dash nh. And somewhere on the mapserver.org website that I'm one of the maintainers of, it's there. But it's kind of hard to find. But that's, that's one of the things you want to just keep in your back pocket as you're working through problems.
04:40
Definitely. Because like Mike said, we do list the URL where it's located. Right. But it's kind of, right, true. But it's kind of hard to find sometimes. And anyway, yeah, getting away from Apache, all the front-end stuff, and getting the error right at, or crash, right at the command line is useful.
05:05
Another thing you can do with debugging is set your debug logs to go to various different locations. You can use the Apache log, which is a default. Or you can also specify specific error files that you want your log files to go to. That's something that we do.
05:20
And if you're a Unix user, and you're getting into actual crash situations or other things like that, you can run Map Server in GDB, the new debugger. You can set breakpoints in the code and generate back traces to see where a crash might have occurred. And that's something that the developers really need to see when you when you do have a segfault or something like that.
05:45
Seeing the back trace can let us know exactly where the crash is occurring, and we can diagnose something easier. So it's good for submitting bug reports. Right, and I actually, I maintain MS4W, so I do a lot of Windows builds, especially recently.
06:04
Just recently? Yes, I mentioned Windows here at Phosphor G. Anyway, yeah, so there is some important steps. I mean, maybe I'm the only one in the room that this applies to, but if you are building binaries for an open source project on Windows, there's ways to debug that.
06:28
I've listed a few things, like many of you already heard of dependency walker. It's been around for a long time to see if you're in DLL hell, as they call it, missing a DLL, got conflicts. There's also C++ redistributive problems.
06:43
That can usually, you can usually, those dependency walker and process explorer will help you through that. Anyway, but it's not fun. And that's probably a reason to avoid using Windows in the first place, isn't it? Debug level two, or higher, but debug two is where it starts to come in, is also good for not just
07:05
finding problems, but also for documenting performance of your system, because it provides layer timings. So what we do is we have a variety of map files and requests that we make of certain areas at various resolutions, and
07:21
periodically, whenever there's changes to our map file, we rerun it, get our generated layer timing, so we have expected timings for each layer at various zoom levels over areas of interest. And then when we have performance issues that are reported by clients, we can compare that to what our expected norms are. And then we know that there might be maybe a server issue, a problem with a spatial index,
07:44
some kind of data corruption issue, or somebody's changed some symbology that's, you know, causing a performance issue. So that's something that we do to keep track of performance. That's you again. I guess it is.
08:01
So a new feature that was added, I believe in the 6.4 release, is the ability to secure OGC services by IP address. This is something you want to do when you have services that you only want to make available to certain clients or certain groups of clients.
08:21
It can be done at the map level, but typically this is done more at the layer level. You can specify these addresses by range, by external files, and it does allow IPv6. You can even just control services at certain levels. So you can IP restrict WFS, but enable WFS for every, or WMS for everybody. So you can just allow certain
08:48
protocols to certain users. Say they, you want to provide WFS services to your authenticated clients, but WMS services to everybody. And the other thing that you need to do is block access to the map server CGI
09:04
interface when you want to restrict these OGC services. Yeah, this is an oldie, but a goodie. And we added this in recently. I just kind of fell across this for one of my clients in like the last couple months, and
09:23
I never even knew about it. It's been in there a long time. Yeah, yeah, yeah like like RFC 30-something, and we're at a hundred and something. 117. So it's been around a long time. Yeah, you've used it? Have you used it?
09:45
Okay, well, okay. You're a power user. You don't count. Anyway, so this is, it's kind of cool. I think Hobo added. Power, yeah. So right, it says here early times, underused.
10:02
So it's, the idea is that a map server will hopefully use a GDL-OGR library to read the projection, to get the projection information. If you're using a GDL-OGR connection to your data. Right, right, and that's the second point. The key thing is that you must use an OGR slash GDL connection. So that second point there, connection type OGR, that's for vector layers, right?
10:27
So make sure you do that before you try triggering the projection auto. Yeah, so I think I, we have a couple, couple of, that last point is very important. I discovered it recently that it doesn't work with external world files.
10:43
So a little bit of a caveat here. Using the connection type OGR can impose some performance penalties if you have large data sets. The native drivers are definitely faster than going through OGR. So that's something to be aware of. But you know, we didn't even plan this, but you mentioned that. I want to elaborate on that. Because I think that's a power tip in itself, is just going through OGR.
11:07
Like even if, dare I say it, you have a shapefile and something's not working. We, you and I have solved things by going through connect, like an OGR connection, right? It sometimes helps you isolate where the issue might be. Right, so keep that in your back pocket, too. Like,
11:24
yeah, you can use the native, I guess you'd call it. Yeah, native map server connection, or run it through connection type OGR, if you're dealing with a vector. And, and you might, you might find a difference there. And then you can take that to developers. Yeah, sorry.
11:43
Yeah, that's, that's just an example with a SQLite database, sorry. Yes, so it's just simple. You can see projection auto. Yeah, and it works. It should work. Rasts are very simple. It's just exactly, just projection auto. So map server will read the coordinates system from the,
12:03
from the TIFF file through GDAL. Assuming it is correct. Yes, right, right. And again, remember it doesn't, it doesn't, map server won't read the external, right? So if you have a TIFF, a world file for that TIFF.
12:20
Right. Mike. This is a new feature that was added at Map Server 7. It was an RFC that was added for Inspire support done by Evan Ruall. WFS 2.0 is now native in Map Server and one of the neat things about WFS 2.0 is paging is supported by default.
12:43
We had added paging support at WFS 1.1, but it was a non-standard extension syntax. It's definitely, it's standard syntax in WFS 2.0, and it does support sorting as well, so you can specify sorting keywords. And
13:02
WFS 2.0 also supports time support. So you can make time dependent spatial queries with WFS 2.0. Another kinds of things you can do are stored query supports. So you can set up WFS queries on the server side and refer to them by names and pass parameters to those names.
13:23
I'll give you an example here. And like I said, it was a base for Inspire support. So here's a couple examples of some WFS 2.0 queries, where here's an example with a time period query, where you have a begin time and an end time, and
13:43
this also has a sort order on it. So these your data has to have actual time data in the database or whatever your back end is in order to do that. But it can do time-based WFS support. And here's an example of
14:01
creating a stored query. So I specify my query name. I can point to some XML file that contains that query, and I can define some parameters for that query. And then, rather than having to specify the entire WFS filter, I can specify just some query ID,
14:21
some name, and then specify some parameters to be passed to that. So it's a way of making WFS queries a lot simpler, especially when you're using similar kinds of operations repeatedly and just want to change a few parameters.
14:40
And this is a kind of a sub-pro tip on WFS 2.0. In the Map Server 7 release, we've refactored how Map Server handles how it handles spatial filtering and attribute filtering. So everything gets pushed now down to the native driver level at the lowest level. So
15:05
this wasn't too much of a problem when you were doing WMS because the BBox filters were implemented natively. So when you made a request for data in your shapefile or your database, PostGIS, whatever, it would limit that filter by the BBox.
15:21
But when you're doing that with WFS, you may not have a bounding box. So any attribute filter or other kinds of filters that you specified at WFS, that was done at Map Server. So all the data would be brought back from your database or from your shapefile, and that would be filtered in Map Server. Now those spatial filters are actually pushed natively to whatever your back-end is.
15:45
MySQL Server, PostGIS, OGR, whatever it is, and implemented at the data driver. So if you're doing a database query and you specify, you know, a time filter and some attribute filter, those happen at the database. So the speed-up is tremendous.
16:05
So if you're doing WFS queries on some kind of spatial back-end and you're not doing them with BBox filters, you definitely want to be moving to Map Server 7 and you use the new native drivers that push that to the back-end.
16:22
Yeah, and this may be the most significant, I mean in my mind, the most significant addition to Map Server 7. And it delayed Map Server 7 for quite a while because one of the drivers needed, right? He was kind of waiting on one. Well, not just one, a series of them, it was because it was a lot of back-end development. And Mike here, to toot his own horn, Mike actually
16:46
did a lot of work with the Oracle driver for this. For this release? For this release. Okay. Yeah, we're gonna go quicker now, I think, because it's almost time. So HTML-based legends, many, many already know this trick. This has been around for a while as well.
17:04
But I think nowadays is sort of now again underutilized. And this is for CGI only. Give it a try. I mean, I documented this so long ago, like for Map Server, I think it's very powerful. We just add a template into your map file, point to a template,
17:27
and you get a, you know, an accurate legend, HTML legend. And just quickly, you can't even see that, but there's some syntax for HTML as well as JSON. Yeah, those are just some examples of what the templates might look like to get HTML legends in a variety of different formats.
17:47
Yeah. So pretty powerful. There's lots of formats to learn. Just give it a try. Content-dependent legends, do you want to do that? Is it? Yeah.
18:02
No, it's not yours. Okay, I can talk to that. Okay, so another recent addition to Map Server between the 6.4 and 7.0 release are content-dependent legends. The ability to only show the information within the actual view. So when the get legend request is
18:21
generated, you have some additional parameters that you need to add to your request, including the bbox, width, height, and SRS, and then it will only return the get legend request that includes the symbols, the feature types that are present within that field of view. So it doesn't give you the entire legend.
18:43
So if you had 20 layers that you're requesting, but the field of view actually only showed two layers, your get legend request will return an image with only two entries in it, rather than all 20 layers. But just to be aware that doing this has to do more work because it has to make a request into
19:01
your actual features, see which ones are there. It doesn't render them. It's using that to generate the legend, but it's filtering them based on that. So if you don't do that, if you're just choosing a full legend, it can just go through the layer list, generate the legend preview, and return it very quickly. So there is some expense to this.
19:22
And that's yours. Yeah, we're definitely geeks. We have a lot of text and code, so we're not so many fancy. Yeah, so advanced blending modes. This is a cool thing that was added right near the end of the release, right? Very end. So it's really not, I don't see it out there a lot, you know.
19:42
This is bleeding edge. This is darn bleeding edges, and I was just making these screen grabs a half hour ago, but yeah, with Map Server, but yeah, so it's really bleeding edge. I really haven't seen it out there. But yeah, so
20:00
there's a new block in the map file, like a composite block, so layer and then composite, and in there is a new parameter called composite comp op parameter, and there it gives you something like 15 to 20 different operations where you can actually blend like two layers, right? So think of things like
20:24
tail shades, you know, the old opacity trick. Yeah, yeah, so the so the idea here is you you'd have things you can do with GIMP. Kind of things you do with GIMP, right, and you can have multiple composite blocks for one layer.
20:42
Yes, so you can throw an overlay, you can throw a light and some sort of contrasting, yeah, so it's kind of, yeah, it's there. Right, right, yeah, so I mean there's my attempt at, yeah, so right, so I've taken like a shaded, shaded relief, and then like some local
21:03
colors, RGBs, and then I've just done an overlay with with that syntax. So it's pretty simple. So it's all done by Map Server. Yeah, I don't know. It's pretty cool. Yeah, yeah. It wasn't officially released in Map Server. It was in the code in Master,
21:24
but it just been released in the 7.0 release in July. Yeah, yeah, there were other tricks to do that too, but this, yeah, this is this new composite trick. Well, that's the kind of, it's very performant this way. Yeah, super performant. Yeah, that's a good point.
21:41
Yeah, so you like you said you can, yeah, okay, so look at that. We're all done. That's it. See you next year. Guys, just one question because I need to have Jeff on the annual meeting in a few minutes.
22:05
Yes. It will start in, by the end of this session, so just one question. Yes, please, Luca.
22:22
Like you mean, yeah, I mean, why would you do that? Is that what you're saying? Like why wouldn't you just combine the file and create the, the Yeah, oh, oh projection auto. It's probably well.
22:45
How about they fix that? Yeah, yeah, that's an older issue. Yeah, that was fixed back in Map Server 5. Yeah, that's fixed. Everyone hear that, but that's a good power tip too. Well, we used to be a power tip,
23:01
so if your EPSG file, if the projection that your app's always using was way at the bottom, Map Server would have to go line by line and read that and there was no caching. Right. Now there's caching. So maybe there might be a hit at the beginning. I don't think so. Yeah, but but to answer your question, we haven't really benchmarked
23:22
projection auto, but I've never noticed in my testing any. I think there's not really a performance hit. There's just more potential to go wrong if your spatial information file is not correct. That's the only reason. Exactly, yeah. Yeah, and then you, yeah, and you, you're throwing that into OGC services
23:42
and you really want to take that risk. If you know the source projection, why not list it there? Anyway, you know what I mean? Yeah, all of them.
24:01
There really isn't too bad an issue with really large map files. We, we typically run 30 to 40 thousand line map files. Now these are ones that have a lot of include files and sometimes includes files that include files, but we don't see much of a performance hit
24:21
with, with really large map files. Now that being said, we do tend to run a fair amount of RAM on those systems. So the, the stuff stays parsed and we do run fast CGI. So that is an issue, but I, even with Silver Ed,
24:44
it's been, I wouldn't say it's fixed, but it's just kind of been improved over the years. And that, you know, maybe if you had two hundred thousand lines or five hundred thousand lines, you might get into an issue, but at, you know, forty to fifty thousand, we haven't seen anything.
25:05
So, thank you very much. We really need to go to the annual meeting. Please join.