Past, Present, & Future of MapProxy
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 95 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/15555 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Nottingham |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSS4G Nottingham 201355 / 95
17
25
29
31
32
34
48
50
56
58
68
69
70
82
89
91
00:00
Proxy serverCore dumpSoftware developerOpen sourceBit error rateCache (computing)AreaSet (mathematics)VarianceClient (computing)Server (computing)Ultraviolet photoelectron spectroscopyWeb serviceMappingWide area networkWorld Wide Web ConsortiumPersonal digital assistantSource codeTemplate (C++)Uniform resource locatorMaximum likelihoodCuboidPolygonOracleShape (magazine)Demo (music)Information securityInformationCodeXSLTSoftware testingModellbasiertes TestenScale (map)Asynchronous Transfer ModeComputer-generated imageryEstimationCAN busNewton's law of universal gravitationLibrary (computing)Line (geometry)Probability density functionElectronic mailing listOnline helpMaxima and minimaLocal ringDigitizingSinguläres IntegralOverlay-NetzZugriffskontrolleLimit (category theory)Type theoryBound stateComputer networkOperator (mathematics)Computer-aided designCASE <Informatik>State of matterStandard deviationAuthorizationDifferent (Kate Ryan album)Axonometric projectionSingle-precision floating-point formatToken ringPublic key certificateCodierung <Programmierung>Exception handlingMaizeMathematicsDefault (computer science)Code refactoringDependent and independent variablesGateway (telecommunications)Interface (computing)Java appletProcess (computing)Data modelTelecommunicationSynchronizationPriority queueStructural loadBit rateAdaptive behaviorComa BerenicesCASE <Informatik>LogicMathematicsTexture mappingTessellationRow (database)Factory (trading post)Web serviceMereologyProxy serverDependent and independent variablesSoftwareMedical imagingOperator (mathematics)Point (geometry)Information securityProcess (computing)Order (biology)Module (mathematics)Macro (computer science)CodeDifferent (Kate Ryan album)Standard deviationQuicksortLimit (category theory)Physical lawNetwork topologyImage resolutionSoftware frameworkComputer fileMultiplication signSet (mathematics)Endliche ModelltheorieMixed realityAreaData storage deviceCondition numberLevel (video gaming)CuboidMatching (graph theory)Scripting languageSymbol tableMultilaterationBoss CorporationFunctional (mathematics)Information privacyCodeSoftware testingInformationUtility softwareFile systemWhiteboardState of matterProjective planeBoundary value problemRevision controlClient (computing)Physical systemSatelliteMetropolitan area networkMessage passingDatabaseForestWordRight angleCartesian coordinate systemUniformer RaumComplete metric spaceBlock (periodic table)AuthorizationStructural loadLocal ringInstance (computer science)Server (computing)Online helpText editorSpacetimeArchaeological field surveyScaling (geometry)Real numberMultiplicationDemo (music)Open setLine (geometry)CausalityMilitary baseBitFerry CorstenReading (process)Procedural programmingPower (physics)MappingEmailOpen sourceNeuroinformatikWeb pageInsertion lossFeedbackValidity (statistics)Repository (publishing)DistanceBuildingWebsiteDigitizingStreaming mediaPresentation of a groupCommitment schemeConfiguration spaceWorld Wide Web ConsortiumGreen's functionCache (computing)Overlay-NetzSource codeImplementationInternet service providerTesselationComputer-generated imageryLoginFile formatProduct (business)Query languageSuite (music)DemonSynchronizationAdaptive behaviorBit ratePriority queueProgramming languageProcess modelingVolumenvisualisierungZoom lensCore dumpDirection (geometry)Front and back endsDigital watermarkingSoftware developerPolygonUniform resource locatorGateway (telecommunications)Interface (computing)Parallel portWater vaporMiniDiscComputer animation
Transcript: English(auto-generated)
00:00
I'm really happy to introduce you to Omidyakomnica, he's one of the founders of Omidyakomniscale, a company leading an open talk here at Stux in Northern Germany. And yeah, I think you're presenting something like your baby-old happy format.
00:22
So he's talking about past, present and future of the network team and I'm sure there will be a future for them. Hi, I'm Oliver Tonnefer, I'm the core developer of MapProxy and as Till said, I'm one of the co-founders of Omidyakomniscale,
00:41
we're a small company in the northwest of Germany and we do a lot of open source and open street map stuff. And today I want to talk about the past, the present and the future of MapProxy. And the past. Well, how we started with MapProxy, it was the end of 2008 and we had the idea for MapProxy.
01:08
We knew that tile caching is past and that it will be the future, but we also knew that the government requires WMS services. And our idea was we just combine the tile cache with the full WMS and make a product out of it.
01:30
And this is how the first version of MapProxy looked like as a schema. We have from one side, we have clients that communicate with MapProxy and MapProxy itself communicates with other WMS servers or tile servers.
01:49
And the first features of MapProxy, of course, was that it supported tile requests. So clients make requests to MapProxy and MapProxy delivers tiles.
02:01
When these tiles are missing, MapProxy would request the source service to fill the cache. We also supported WMS from the first version on cascaded requests. So MapProxy just forwarded the request for the same exact bounding box and listen, hey, and projection to the source.
02:23
And MapProxy would return that image. But we also supported responding to a WMS request from cached data. And this is quite unique for MapProxy. And it does it by maybe you can see here just a map with these should be tiles.
02:49
And when a request comes in, MapProxy just combines multiple tiles that are all affected by that request. We'll create one image, merge the tiles together, scale them to the
03:05
exact resolution that was requested by the user, and it even supports reprojection. So when your cache is in one projection, but the user requests another projection, it does the reprojection.
03:20
And it does it outbound, inbound, and on the fly. So when you have a cache in one projection, it can reproject outbound data, but it also can reproject on the inbound. When you want to build up a cache in one projection, but your source doesn't support it, it can reproject it then. And on the fly for cascaded requests.
03:44
And we used MapProxy in production since 2009, but it was only used by us. And software only gets better when you have more users, when you get more feedback. And when people with other use cases come to you and say, it would be great if MapProxy could do that.
04:04
And so we open sourced MapProxy, and all the features are listed where in our first open source release. We started with version 0.8, and it was in May 2010. And from then on to the present, we have now six major releases in the last three and a half years.
04:28
And we added lots of features, most of the time to make things easier, and also to solve completely new use cases.
04:41
And now I want to highlight some of the features that we added in the last three and a half years. One feature was the 0.8.3 release, that we now supported basically any tile source. You can just create your own, write down your own URL template, and you
05:02
can access any XYZ tile source, or a WMSC that requires a fixed parameter order. We also improved the seeding tool. Seeding is a process of pre-generating the tiles, so that your cache is not empty at the beginning.
05:24
And since then, the seeding tool supports parallel seeding, so MapProxy makes multiple requests to the same source. And we also support a feature that we call coverages.
05:40
And with that, you only seed the required areas instead of the complete bounding box. And you can see it in this image. If you want to seed the UK, and you just would use a bounding box, you see that you're seeding a lot of water areas in neighboring countries.
06:00
And with MapProxy, you can just load up a polygon file, a polygon, and just seed within that polygon. And this makes seeding a lot, a lot faster, and you're saving a lot of disk space. And you can load the polygons from GeoJSON files, shapefiles, even databases like PostGIS and Oracle, all thanks to the OGR library.
06:28
With 0.9, we added MultiMapProxy, which is a tool to have multiple MapProxy configurations run all in the same instance of MapProxy.
06:42
So if you want to run different WMS services, you don't have to start up multiple MapProxies. Since then, we support coverages also for sources, and we'll show that later. And we also added a demo service. Since then, you have the ability to easily check all your layers that you've configured with a simple OpenLayers client.
07:10
With the 1.0 release, we added the security API, where you can write your own functions to define what a user is allowed to do.
07:22
And I will also show you, go into detail that in a minute. And MapProxy supports WMS feature info, and you can optionally add XSL scripts to transform the feature information. So if you have two different WMS servers and the format is different of the feature info, you can write your own scripts to unify that format.
07:49
Then we added WMTS support and MapProxy utils, which is a tool that combines multiple subcommands.
08:00
And one subcommand is the serve develop. And with that command, you can easily start a testing server for MapProxy. It will check if your MapProxy configuration changes and will then just reload it. And you don't have to deal with deployment in Apache, etc.
08:24
And we also support, since then, MapNIC and MapServer directly. So you don't have to set up a MapServer inside Apache, for example. With the next releases, we added a few new cache backends.
08:42
Before that, MapProxy always stored the tiles on the file system as simple files. And yeah, we added MBTiles and cacheDB backends. We added more utils, for example, the scales tool that allows you to convert between scales and resolutions, which is quite handy when you deal with MapProxy configuration.
09:06
With 1.4, one of the larger features was the clipping of requests in the security API. And yeah, I will show you that also in a minute.
09:20
With 1.5, we supported reprojection of tiles on the fly. So you have tiles in one projection, MapProxy can transform them into another projection. We support mixed image mode, which is useful for aerial images, which you want to store as JPEG.
09:44
But at the boundaries, where you don't have aerial images, where you want to have transparency, MapProxy will store these tiles as PNG. And even more MapProxy utils, like the grid tool, which shows you how many tiles are there in your grid for each zoom level in your grid definition.
10:08
And even how many tiles are within a coverage that you defined. 1.6, we will release that last week, I think. We added more cache backends and also the decorate image API, which allows you
10:25
to write your own code that can manipulate the images that goes to the client. You could add watermarks or something like that. So now we are at the present.
10:40
And yeah, lots of features. We now had nearly 2000 commits. The changelog lists 260 bullet points. I just highlighted the most important. And so you might ask, this must be a lot of code, but MapProxy itself is written in Python.
11:01
And Python is a very powerful programming language. And so the code base is relatively small. It's just a bit over 30,000 lines of code. And on top of that, we have nearly the same amount of code for tests, which I think is really great for a software like that.
11:21
We have a lot of low level tests and which we also have a lot of high level tests, system tests that take a complete MapProxy configuration. Start up a MapProxy, make requests to it. Check if the response is a valid XML document.
11:41
If everything is valid and it also checks the request MapProxy makes to source requests. And we have a Jenkins instance, which runs all tests of each commit. It's also integrated into our GitHub repository. So we get a warning when someone makes a pull request that breaks the bill.
12:07
And you see the small red green marker and everything is fine. Help documentation now is a hundred and twenty pages. When you get that as a PDF, we have a mailing list.
12:24
You can get support. There are now a lot of local open source companies that use MapProxy, that have knowledge in MapProxy. And of course, we do support for that. So the interesting part is what can you do with it?
12:41
And now I have some use cases of MapProxy. Well, of course, it is a tile cache, so you can use it for simple tile caching. It's used in the German Broadband Atlas, for one example. The nice thing about the tile cache of MapProxy is, of course, the seeding tool.
13:06
You can accelerate existing WMS services. This is used a lot in Germany, where we have a lot of old and rusty and slow WMS servers. And they want you to accelerate them.
13:26
As I said, you can transform tiles. And this is used by a few OpenStreetMap contributors. And they have access to OpenData tiles from a local tile server. But these tiles are not in the Web Mercator projection.
13:43
And they use MapProxy to get the data into a JOSM, an OpenStreetMap editor, so that they can digitize roads and buildings. Another example is, you have your own WMS server.
14:03
But unfortunately, that WMS doesn't support the Web Mercator projection. But you want to overlay your WMS server with OpenStreetMap tiles, Google Maps, etc. And you can use MapProxy to re-project the data.
14:22
Now the security API. MapProxy can restrict access to your source services. And it does this with a security API. So you can write your own code that connects to your own user database, for example.
14:42
And your code can decide what the user is allowed to do. You can limit services, like WMS or tile request. The request type, like feature info. And the layers, but also the areas. And for example, you can say user A is allowed to see layer X, but only within the boundaries of Germany, for example.
15:08
And everything else is clipped. And in this case, it's made transparent, so that you can see the layer below. And this is quite powerful. Satellite provider uses this to restrict the access to paid customers.
15:26
And it also works with tiled requests. And last but not least, one larger use case. This is from an actual national network operator in Germany.
15:42
And their customers, they have a network on the last mile to the customers. And that requires them to have access to official cadastral maps. And in case of Germany, we have 16 federated states.
16:04
And so we also have 16 cadastral WMS services. So WMS, it's a standard, so it's quite easy to use it. That's what you would think. But when you look in detail to all these WMS services, you will see that the cadastral services are mostly not free.
16:27
So you have to have some user credentials. And they all use different authorization methods. Some of the WMS implementations are just broken. They don't return any EPSG codes.
16:41
They return invalid bounding boxes. They don't behave like the OGC standard says. And there's not a single projection that's handled by all, etc. So when you want to make these WMS services available to hundreds of in-house users, then yeah, good luck.
17:04
But you can use Map Proxy for that to unify all services. You can have a single URL for all cadastral maps. And you can use the coverage feature for all your sources. So you can say that this source only delivers data in this region.
17:27
And when a request comes in for this region, Map Proxy will only make a request to that server and not to the other one. The authorization is hidden from the user. The capabilities are valid.
17:41
All sources are transformed to a single projection. And we can even make the background transparent for services that do not support that. So make one WMS available to hundreds of in-house users, that's easy. Okay, that's with the present in the use cases.
18:05
I now want to give you a quick look into the future. Now 1.6 is out, so we're now working on 1.7. We don't have any larger features planned because we already are making plans for the 2.0 release.
18:27
And we will make some backwards incompatible changes, use it to refactor some parts. We will also support Python 3. And we will add a new background service.
18:43
We will support Python 3. We will skip the 2.5. And okay, new background service. This is the last point I want to make.
19:01
First I have to explain the process model of Map Proxy. Map Proxy is request-based. The request comes in, one process or threat deals with the process and the response goes out. But as soon as the response is out, you can't do anything anymore because the next request comes in. And this is given by the web service gateway interface that Map Proxy implements.
19:25
And it's similar to how CGI and Apache modules and JavaScript work. So limitations are that you don't have any long-running processes. As soon as the request ends, your process stops.
19:41
And you can't really communicate between requests. There are workarounds. Map Proxy uses, for example, an external tool to have a long-running process to see tiles. And Map Proxy also uses file logs for some synchronizations. So that not two processes make the same request to the source.
20:03
But it's all limited. If we had a long-running background service, we could implement all these features. We could add something like a priority queue so that we could limit the load to the sources.
20:24
One process, we have two running queries to the source. We're not making any more, we're just queuing that. We could add seeding without having cron task, etc. We could rate limit requests and implement adaptive pre-seeding, etc.
20:44
So a lot of features that would be possible with that. We have a Map Proxy render daemon that's available at GitHub. It works with 1.6. We already use it for two projects. It's not officially announced.
21:02
And with Map Proxy 2.0, we will add a Map Proxy daemon. It will be an optional feature. And it will not include all these features. But it will be the framework to implement all these features in the future.
21:26
And summary, Map Proxy is more than just a tile cache. There's a steady growth of features that we hope that we test them all well with all the test suits. And there's more to come in the next releases in the next years.
21:44
And that's it. Thank you very much.