MapServer Project Update - Introducing Version 7.0
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 188 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/31712 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer | ||
| Production Year | 2014 | |
| Production Place | Portland, Oregon, United States of America |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
State of matterLine (geometry)Repository (publishing)Right angleCommitment schemeElectronic mailing listSoftware developerNatural numberTerm (mathematics)DialectQuicksortReading (process)Self-organizationSoftwareStability theoryMultiplication signNumberService (economics)Group actionTexture mappingUser interfaceProduct (business)Server (computing)Inheritance (object-oriented programming)ImplementationOpen sourceOperating systemApache ForrestCapability Maturity ModelSheaf (mathematics)Web serviceNapsterSatelliteError messageClosed setGrass (card game)Computer programmingSource codeBitFile formatForestField (computer science)Domain nameSlide ruleAreaEvent horizonFunction (mathematics)Web 2.0Cartesian coordinate systemExtension (kinesiology)Computing platformDescriptive statisticsWebsiteGeometryContent (media)BuildingTesselationEmailPoint (geometry)Different (Kate Ryan album)Level (video gaming)Stack (abstract data type)Scripting languageCodeComputer animation
07:33
Population densityMultiplication signCodeBitPoint (geometry)State of matterTransformation (genetics)Vector spaceEstimatorProduct (business)CodeTwitterKernel (computing)Software developerMathematical analysisSurfaceFluid staticsService (economics)Server (computing)AutocovariancePolygonMetric systemLine (geometry)Term (mathematics)Software testingType theoryQuicksortLatent heatFilter <Informatik>Software suiteResultantView (database)WeightQuery languageAttribute grammarAdditionEvent horizonRepresentation (politics)Set (mathematics)Form (programming)Installation artVector potentialConfiguration spaceLinear regressionTexture mappingImage resolutionWebsiteImplementationMereologyRemote procedure callInformationSoftwareMathematicsColor spaceCommunications protocolRevision controlOpen sourceContinuous integrationInterpolationLeakoutputTouch typingUser interfaceParameter (computer programming)Semiconductor memoryDivisorInteractive televisionSeries (mathematics)Source codeINTEGRALCuboidCategory of beingVolumenvisualisierungMetropolitan area networkBeta functionRaster graphicsComputer animation
15:00
Position operatorPoint (geometry)DivisorVolumenvisualisierungIntegrated development environmentElectronic mailing listLogicInformationConstructor (object-oriented programming)Texture mappingGraph coloringProcess (computing)Shape (magazine)Different (Kate Ryan album)AdditionLibrary (computing)Scaling (geometry)Server (computing)Functional (mathematics)Direction (geometry)PixelSet (mathematics)Single-precision floating-point formatCodeBuffer solutionExpressionGeometryComputer configurationConvex hullCategory of beingCyberspaceNumberString (computer science)Kernel (computing)Forcing (mathematics)Type theoryModel theoryWeightFreewareGame controllerQuicksortMereologyDatabase normalizationFilter <Informatik>Level (video gaming)Event horizonFunction (mathematics)Graphical user interfaceFormal languageGreatest elementScripting languageColor spaceObject (grammar)Context awarenessOperator (mathematics)TesselationCache (computing)Bounded variationAreaAsynchronous Transfer ModeDialectPhysicalismMultiplicationStandard deviationBitSound effectSummierbarkeitMoment (mathematics)Goodness of fitMetropolitan area networkConjugacy classComputer animation
22:28
Scaling (geometry)Direction (geometry)TesselationStudent's t-testServer (computing)Texture mappingComputer animationDiagram
23:02
Computer fontFormal languageINTEGRALMereologyRight anglePresentation of a groupAcoustic shadowSource codeLibrary (computing)NapsterDrop (liquid)Sound effectFront and back endsLinear regressionTexture mappingVector spaceDatabaseServer (computing)ArmVariable (mathematics)Multiplication signTemplate (C++)Translation (relic)Fuzzy logicFilter <Informatik>Information securityCategory of beingInformationTwitterCASE <Informatik>Computer fileExterior algebraSequelSoftware testingSoftware developerIntegrated development environmentSet (mathematics)Process (computing)Channel capacityMassOperator (mathematics)Term (mathematics)ExpressionSoftware maintenanceSocial classLogic gateReading (process)Code refactoringDemo (music)Genetic programmingVolumenvisualisierungComputing platformRaster graphicsDirected graphTesselationWriting
Transcript: English(auto-generated)
00:00
Well, thanks for thanks for coming. My name is Steve lime. And I work for Minnesota, our state of Minnesota and their Department of it services, affectionately called minute I'm housed at the Minnesota Department of Natural Resources, resources, though. That's where my background is. I'm a forester, sort of by by trade. And I'm here representing the map server project.
00:28
I'm the PSC chair, and have been so for for for quite some time and we have a bunch of other folks here that are with the project. Get a few slides in and I'll, I'll point them out. And I'm here to sort of provide a couple things. One is project
00:46
update. And then the second is to introduce folks to maps over seven, and which is the newest release, or soon to be newest release here. Come Saturday, I hope. And
01:02
So run through some of the features a little bit about a little bit about the project, hopefully leave a little bit of time for question questions again, some of the other technical leads are here and we'll be able to answer maybe different questions and I, I can answer. So hopefully folks know what it map server is it's changed over the years. And I think
01:26
that the best description these days is that it's a platform for for publishing spatial data web services and and to a lesser extent applications to the web. So it's really about publishing spatial data maps.
01:40
Geoservices to the web and doing so in a very robust manner and in a, in a very high performance manner. So it's, it's a very targeted piece of software. You don't have a whole lot of There's not a website, you can go and publish 1000 or 100 million tiles or things like that. It's really sort of that low level tool for building building content that can be distributed over the web.
02:05
It's an open source project. It's embraced a MIT x 11 style license. So basically, you can do whatever you want. Just give credit where credit is do support it on all eight all major operating systems and Windows, Linux.
02:20
Raspberry Pi. Solaris, yeah, you can compile it and run it just about anywhere. It's an OS geo product and it's, it's also both a product. So there is something called map server that you can download and install. And it's also a project in it. It's an umbrella organization or
02:40
Thing that houses, a couple of other pieces of technology as well. So very brief project history. This is a and one of the older open source geospatial projects that are out there. I think probably only grass is is been around longer, quite a bit longer, but
03:01
Depending on your definition of one thing started. I've been working on it for close to 20 years, probably in 1994 1993 era. It began as a project at the University of Minnesota. We with some with some NASA funding to try and deliver satellite imagery to field foresters and
03:22
it was released that project was called for net and it was released as a as an open source project or product at the end of that program. Program. In 1999 so that was sort of the first public release of the of the software. It was in
03:44
use in a few different spots, probably in the 1996 1997 area. So it's been around a long time. I don't think it really got its legs, though, until about 2000 which is when Dan Morrissette and the folks at DM solutions.
04:00
Recognized that maybe there was some potential here and it was better to collaborate than to create something brand new. And so that's when you started seeing things like the windows builds and PHP map script and a lot of things along those lines. So that's when the project, the project really got going. I was one of eight OS funding always geo projects in 2006 and then in 2012 we this is when this transition from sort of a product to a
04:30
project happened and we added a couple of important tools to the map server stack. One of them is map cash and you're going to hear about that in
04:40
63 minutes or something like that. And then Tiny OWS, which is a WFST implementation that that is backed by PostGIS. The again, I'm the chair of the steering committee. We have 13 other members. So let's see who's here. Jeff McKenna, Mike Smith, Thomas Bonfort, Daniel Morrissette, Howard Butler.
05:07
Yep. And Olivia is here. He's at the conference. I don't know if he's here today. The important thing to note here is it's it's pretty multinational group and no two folks are with the same, same organization. So it's there is no parent map server company or no commercial entity that
05:27
sort of is an umbrella organization to it. So in some respects, it's relatively unique in that regard. There's no board of directors or stockholders to answer to, which is which is kind of nice. In terms of a community, the map server user's mailing
05:49
list is over 1800 subscribers and we've got 14 or 400 or so on the developer list. There's 30 folks with commit rights to our GitHub repositories.
06:01
Those numbers have remained relatively flat over the last few years. The last time I checked was 2011. It hasn't really gone up or down appreciably since then. I think that speaks to the stability of the software and things like that in general.
06:21
We've been an active community in terms of conferences, both hosting, attending, presenting, maybe not as active as some organizations, but you'll always see map server faces no matter where you go in the open source GIS world. So we've hosted map server user meetings, which were sort of the forerunners to this event. The year before OSGEO was created,
06:48
we hosted something in Minneapolis called Open Source Geospatial and then have been an active member of some of the Phosphogy events. And then also real active in some regional events in Canada, Brazil, Europe, and other places. And one of the neat things I think this
07:08
group or this project does is we're a real active participant in the Sea Tribe code sprints, which are an annual event and a great way to sort of get together when you've got such a dispersed group, it's nice to get in the same room at least once a year.
07:25
In terms of infrastructure, you can find us at mapserver.org. It took us years to pry that domain name away from somebody, but eventually I got it. The source is hosted at GitHub. We have a continuous integration suite, so whenever anything is committed to GitHub, a series of regression tests, about 1800 of them are run.
07:51
And you can check out what's going on with the latest builds at that URL. And testing really is important to the project. So we have a test suite of about 1800 tests that put a lot of the code through its paces.
08:07
One of the reasons 7.0 hasn't been officially released yet in beta is because I broke nine tests and I need to fix them on Saturday. Version 7, this is the latest, latest greatest. Version 6 was released about three years ago.
08:28
I think we've been aiming for a major release every couple of years. Was that the desire? And 6.4 was released a year ago. And to be honest with you, 6.4 felt like there were a lot of
08:41
major features added in that release and we kind of went back and forth. Should this be 7? Should this not be 7? In the end, of course, it's 6.4. And I think we did that because there weren't any regressions introduced in terms of configuration and things like that. Where we knew some features that are in 7 had the potential for breaking older installations. Not in
09:04
a really bad way. This is all good stuff, but they're a little bit more disruptive at this point. We had hoped to have actually the official release done, but I've been screwing around on one of the RFCs here for the last three months trying to get everything working.
09:22
And so I think a lot of folks have been using some of this code in production for some time now. It's just the latest, I'll point out my mistake in a few minutes. But we hope to get a beta out at the code sprint on Saturday. Finalize some of the tests and so forth. So in terms of development metrics, according to GitHub, only 234 commits in the last year.
09:46
Now that's really not fair because a lot of the RFC work that has been done Thomas did a whole bunch of work with Coverity. They have a service now where you can run static analysis on open source projects.
10:02
So we fixed a lot of memory leaks, other potential issues within the code. But all that work, which could be hundreds of commits, goes into the master as one. And so just looking at your commits probably isn't the best way to do it. The code base itself is around 260,000 lines of code.
10:25
55 man years or person years of effort in it. And this latest in the last year, so you've seen 92,000 lines touched and 35,000 lines removed from the code base.
10:41
So it's a significant addition and set of changes have been made. So now I'm just going to go through some of the highlights on some of the new features. So the first one is support for UTF grids. If you're not familiar with UTF grids, this is a map box specification for rasterized interaction data.
11:08
So basically it's JSON data that is registered with your map at a lower resolution than your actual map so that you can tie events to that information.
11:23
And do things like little pop-ups as you move over polygons and things like that. It's a pretty neat feature. It was implemented as a Google Summer of Code project. Not this year, but last year. So it's been in the code base for a bit.
11:43
And UTF is supported by OpenLayers, Leaflet, Google Maps and a bunch of other client-side mapping pieces of software. It's not yet supported by MapCache, but I understand that that's in the works or soon to be. Thomas can give details on that.
12:04
Another big addition is WFS 2.0 support. Evan Raynault added this. The specs have been out since 2010. And it's been designated as the base protocol for inspired download services.
12:26
So that was what prompted this addition. There's been a lot of changes to your basic WFS request types such as get capabilities and get features. There's a new filter specification that includes time filtering that's been added. And then you've got a couple of new request types.
12:48
One of them is get property value, which is sort of a super simple version of a get feature request. You're just requesting the value for an individual attribute for a feature. And then something that I haven't used myself, but this notion of stored queries, which are, I kind of like to think of them as WFS views.
13:11
Sort of predefined query sets that are available and can be deployed by your server. So it's not necessarily creating new layers, but representations of those layers that have been cached and are available for performance issues.
13:26
It's important to note that even though the specification provides, or WFS 2.0 provides specifications for the ability for remote creation of stored queries and deleting of stored queries, those are not part of this implementation.
13:45
But it adds neat things like support for GML 3.2.1, paging, which actually was part of Map Server 6, but that was an optional addition at that point.
14:00
It wasn't required by the specification. And then of course, sorting of results. One of the neat additions that I think is, well it's cool because you can actually show pretty pictures of it, is the addition of heat map layers, which are, it's a form of kernel density estimation.
14:25
And it's a vector to raster transformation, so you provide a bunch of points, and does it even work on other feature types? So lines and polygons? Yeah. But you can do it.
14:40
And so what Map Server does is then it will generate a raster surface for you based on input. And then you can configure both the interpolation parameters, so how big is your window, what sort of weighting factors do you want to have, what color space are you going to operate in, and then also how is it going to be rendered.
15:01
Well actually that's the color space piece. But it supports things like automatic scaling of data, you can work in both RGB and hue saturate, or HSL color space, you can weight the value of a particular point using expressions, a lot of neat things.
15:22
And this does support tile mode, so you can use it in a tiled fashion, but you can get into some, it seems more natural in a single image mode. Because if you're doing things like global scaling, you could be in an area where there's just not a whole heck
15:43
of a lot going on, yet in another part of your dataset there's gigantic, Thomas had a good example of earthquakes. So if you're looking at a heat map of earthquakes and you're in some place that's kind of boring and there's a big quake in Japan or something like that, you won't see a lot of variation.
16:02
Here's some examples of the heat map output. Now note this is the exact same dataset that's generated all of these images. So the two images on the left, those represent different filter sizes, I think the one on top is a 10 pixel buffer or kernel, and the one on the bottom is a 20 pixel buffer or kernel size.
16:28
And then the two images on the right represent different weighting factors applied to the points to change the amount of intensity that's picked up by the mapping process.
16:42
An experimental feature that I think has some really interesting possibilities is the ability to use JavaScript as a scripting language within Map Server. So we've run into situations where, man, you want to be able to draw a feature or label
17:07
a feature or do something to a feature, represent it based on any number of properties of that feature. And you can only throw so many keywords at something before it becomes really complex. You either need so many expressions or
17:23
you just don't have enough control over the model that you'd like to apply to that feature to decide how to render it. And so rather than kind of force more keywords or more constructs into the code, we thought, hey, what about being able to push some of that business logic into a scripting environment that's associated with each feature?
17:49
So what you do is you can actually define a bit of JavaScript that will take a Map Server shape object. Now you can manipulate and
18:01
look at that shape object, look at its properties, and decide how you want to render it based on a set of business operations within that code. And then you return a style object back to Map Server and it dynamically uses that to render that feature. So here
18:21
comes a shape, we decide how we want to style it, push that style back to Map Server, and off it goes. And that's done with the Chrome V8 engine and it actually works. It's kind of a crazy idea that we talked about at the last year's North American Phosphor-G event.
18:43
And I didn't actually expect map users to do it, and they did. And so it's there as an experimental feature for 7.0. I see great promise in being able to use this in things like in the templating environment.
19:01
So you basically have a built-in scripting environment that is JavaScript, something that everybody knows. It's very high performance. You could embed JavaScript directly into your templates, but have it executed on the server side. There's other options for being able to do geo-processing. So we have some geo-processing capabilities
19:22
by exposing geos operations, buffer, and things like that, convex hull through the Map Server expression syntax. But this would allow you to do even kind of funkier, crazy things with it. So we're excited to see where that's going to go.
19:42
The text label rendering overhaul. This is something that Thomas Bonfort worked on. There were just some inefficiencies within Map Server. I won't say inefficiencies, just some redundancies that were in Map Server where as we added rendering options, each one of those renders had to implement a lot of the text layout capabilities.
20:02
So converting something to a UTF, not UTF, forget it. You take a string and you figure out how you want to lay that out and how do you want to place it and what character is next to what character and how much space to have between them. That had to be done differently every time or for each render. So there was a
20:25
lot of duplicated code, a lot of opportunities for things to go in slightly different directions. So instead of working with strings, all that code was pulled out and a single set of functionality was put in to work with free type and glyphs and very precise positioning information.
20:47
And then those lists of glyphs then are what are passed to the renderers and they can, as Thomas puts it, stupidly just stuff them on or render them on the map. So all that layout work is done in one Map Server function and then these glyphs are handed off to the renderers for actual processing.
21:09
This was also an opportunity to integrate HarfBuzz. This is a text shaper library that can deal with the different contexts of characters in certain languages.
21:22
So in Arabic, a character can be represented four different ways depending on what other characters it's near. We had support via FreeBitty, if I said that right, for some text shaping but that was actually limited to Arabic languages only. So this really expands the multi-lingual capabilities within Map Server and now the engine supports much more precise label placement.
21:56
The end users probably might not, I mean unless you're working in parts of the world where you need these
22:00
or working with these other, these extended languages, there's not a whole lot of sort of end user facing stuff here. But this really sets the stage for additional text processing and placement capabilities moving forward. One of the outcomes was performance enhancements with our label cache processing. Notice these are logarithmic scales in both directions.
22:35
So if you're dealing with millions of labels in one map, it's really a nice improvement for you.
22:48
But surprisingly, I think folks would operate in the one, you can easily have a thousand or two thousand labels when you're dealing with meta tiles. Although Map Server was quite fast in that regard anyway.
23:02
Here's just an example of some, this is a Tibetan language and font. Prior to the Harpas integration, this would have been rendered incorrectly and would simply just not, it wouldn't be readable to folks that spoke or read that particular language.
23:25
GD is dead. And for those that are familiar with Map Server, one of the, it started with basically two technologies, ShapeLib and from Frank Warmerdam and GD.
23:41
And GD hung on for about 17 years, 18 years. And it was just a casualty of the label refactoring. It doesn't have the ability to process glyphs in the same way that we can with AGG or Cairo or some of the advanced renders. It was also becoming a maintenance time sink. And there were just fewer use cases that actually required it.
24:04
So there's just a couple of consequences to pulling it out. One is we lost bitmap fonts, which were fast built-in fonts that were part of the GD library. But Thomas embedded a TrueType font, so you still have built-in capabilities. And we lost
24:27
the support, write support for GIF, although you can still read GIFs for symbol purposes, symbology purposes. And then the one that I've been working on is a filter translation, RFC, which tries
24:43
to push all the heavy lifting off to the databases that support, to backends that support SQL. There was some support for that, but it was only handling the most naive cases via WFS. So if you were just doing property equals value, a lot of that was already pushed off to the backend.
25:03
But dealing with cases where you're doing spatial operations, it's much more efficient to push it to the backend. Otherwise, all those features have to come over the wire and then be evaluated by map server. So that's really what this is all about. And massive performance improvements are possible if you have very large databases.
25:22
I don't know if folks saw the Scribe presentation yesterday. It was up against OpenLayers 3 and a couple other heavy hitters. But it's not technically part of map server, but it's definitely affiliated with it. And this is a UI for managing data and map server map files.
25:44
This is what it looks like. And it's scribeui.org. Is that the URL if folks want to go take a look at it. There's been a lot of attempts at map server writers over the years, and none of them have stuck. And so I'm hopeful that this one has got just the right ingredients.
26:02
This is the editing environment. It uses kind of a JSON-y look and feel, variables, a lot of neat things. In terms of where we're going to go, we talked about the JavaScript support within templates. We want to capitalize on some of the Scribe work that's been done, especially with variables and things like that.
26:22
We're really keen on increasing our regression test coverage. We're probably up close to 70% with some of the Oracle work that Mike has done. We'd like to be able to support alternative ways to style information just besides classes, expressions, and those two.
26:45
Cardo CSS and SLD are obvious candidates. Layer compositing for effects like drop shadows and being able to fuzz edges and things like that is something that Thomas is working on. Lord knows what we need to do with vector tile support. Are we going to be a
27:03
producer? Are we going to be a consumer? I think that will shake out in the near future. And then on the end user side of things, just we need to get a new demo together, something prettier. So how can you get involved? There's opportunities for both users and developers.
27:22
GitHub really makes the source more accessible to folks. We've gotten more poll requests, I think, from not necessarily novices, but for people that wouldn't have ordinarily been able to contribute just through that platform. And of course we're always looking for new folks to come in and help out. So if you know C or C++, we're the project for you.
27:46
So if you have more information, I can be reached at that email address. I don't tweet, so that's the best way to get ahold of me. And of course you can find all that contact information at appserver.org. Thank you.