vts-mapproxy - The 3D Geospatial Streaming Server
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43472 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
MereologyFile formatSummierbarkeitTriangleSoftwareConnectivity (graph theory)Dynamical systemElectric generatorRaster graphicsData conversionVector spaceMobile appState observerPlastikkarteDecision theoryEquivalence relationObservational studyPhysical systemSingle-precision floating-point formatPower (physics)Presentation of a groupWeb 2.0Cache (computing)Remote procedure callInternet service providerMappingMassComplete metric spaceService (economics)Interface (computing)BitOcean currentSurfaceDigitizingCodeEndliche ModelltheorieStreaming mediaComputer fileLine (geometry)Information technology consultingNeuroinformatikFront and back endsMachine visionProxy serverArithmetic meanWeb applicationIntegrated development environmentInteractive televisionDirectory serviceLevel (video gaming)Cartesian coordinate systemThree-dimensional spaceWebsiteUtility softwareConfiguration spaceCASE <Informatik>Multiplication signSlide rulePoint (geometry)Coordinate systemWordData managementRight anglePlanningRoundness (object)Source codeRevision controlInformationWave packetSet (mathematics)Position operatorStorage area networkDirection (geometry)MathematicsGame theoryVideoconferencingSoftware developerChemical equationDebuggerGroup actionExecution unitPoisson-KlammerStandard deviationProjective planeInsertion lossView (database)Formal languageLecture/Conference
07:33
Set (mathematics)Interface (computing)Device driverFunctional (mathematics)File formatVector spaceTesselationType theoryOpen setBitProxy serverAuthorizationConnectivity (graph theory)Front and back endsServer (computing)Stack (abstract data type)SoftwareService (economics)Population densitySoftware developerEmailEndliche ModelltheorieAxiom of choiceFrame problemConfiguration spacePolarization (waves)Spherical capQuadrilateralSphereDiscrete element methodArc (geometry)2 (number)QuicksortLevel (video gaming)VolumenvisualisierungRaster graphicsLatent heatPolygon meshComputer configurationPower (physics)MereologyStandard deviationGoodness of fitPresentation of a groupHierarchySingle-precision floating-point formatInternet service providerNatural languageProjective planeParticle systemPoint (geometry)Bit rateSpring (hydrology)Electronic mailing listMoment (mathematics)Division (mathematics)HypermediaBasis <Mathematik>Library (computing)Source codeECosTheoryPhysical systemForm (programming)View (database)Ferry CorstenUniverse (mathematics)Multiplication signMappingLogicPRINCE2InformationRow (database)Address spaceLine (geometry)SpacetimeLecture/Conference
14:59
Open setCartesian coordinate systemSurfaceLevel (video gaming)Message passingPhysical systemPoint (geometry)Different (Kate Ryan album)Interface (computing)Service (economics)VideoconferencingMereologyMoment (mathematics)ApproximationTesselationStandard deviationMechanism designMultiplication signHypermediaSoftwareDisk read-and-write headInternetworkingPlastikkarteNumberSource codeComputer fileProcess (computing)Neuroinformatik40 (number)Video gameComputer virusDemo (music)Boss CorporationKeyboard shortcutAreaConfiguration spaceInformationNatural languageArtificial neural networkLine (geometry)Machine visionNumbering schemeProxy serverUtility softwareWeb browserSoftware frameworkCache (computing)Repository (publishing)Domain nameDivision (mathematics)Artistic renderingWeb 2.0Directory serviceFile viewerEmailIntrusion detection systemFile systemWeb pageLecture/Conference
Transcript: English(auto-generated)
00:07
So I think we can about just start. So allow me to introduce Andrei Prokhaska, the first speaker of this afternoon session, who will be talking about VTS Map Proxy, the 3D.
00:24
So Andrei, it's yours. Thank you, Vadio. Well, my name is Andrei. I'm currently a consultant for a product-based company called Melon Technologies. It's a big data streaming, deep learning, and computer vision
00:43
company. And it's been recently acquired by Hexagon Geosystems. So we're now part of that group. What I'd like to talk to you about today is VTS Map Proxy. And chances are that VTS Map Proxy is just a foreign-sounding word to use.
01:01
So it's still good to see so much interest in it. And just to make it a little more accessible, I'm going to start with some images. And this is a case study I did for the last plus 4G in Dar es Salaam. It's basically a 3D management model
01:21
of Mars, which comes complete with labels, various data coming from ESA, from NASA, from the Mars Reconnaissance Orbiter, from various sensors. And you get this near-VR experience. This is actually what this was called. And you can still play around with this, actually, because it's online, the URL, which
01:41
will be shown in the next slide. And the interesting point is that this particular case study, sorry, I need to cut this short because the time is limited. But the website is online. So you are free to play around with it. And this particular case study was created almost exclusively
02:04
with a single server-side component, which is running on an off-the-shelf Ubuntu VPS lineup. So you can roll this on your own. It's very limited resources. It's very limited time. You can have a presentation like this. And the particular component which
02:21
allowed this to happen was VDS MapRoxy, the component that we are going to be talking about in the next 20 minutes or so. So first of all, to give you some overview of what VDS MapRoxy is, it's a 3D geospatial streaming server,
02:46
which is providing on-the-fly access to an array of raster and vector data formats, which is serving those data in formats which are optimized for 3D streaming and rendering.
03:00
That pretty much sums it up. As for the features in MapRoxy, what it does essentially is it does dynamic triangular network generation from digital elevation models. That's actually something that was the original purpose
03:23
behind VDS MapRoxy. It does on-the-fly coordinate system conversion for a massive raster data sets. It also provides WMDS service for raster data sets. It can be used as a cesium.js terrain provider if you're into cesium.js.
03:42
And it can work as a VDS geospatial remote pencil provider. If you've never heard about VDS geospatial, VDS geospatial is a mapping stack. It's a complete integrated development environment for three-dimensional map application development, which has been created and has been maintained by my own technologies.
04:04
And it also has massive introspection capabilities, meaning that just by running MapRoxy, you are actually getting this web-based interactive resource directories with embedded viewers, which actually allow you to inspect all the resources
04:20
that MapRoxy is serving and which can be used by very impressive web applications themselves. VDS MapRoxy is part of VDS geospatial. I've just mentioned that, what VDS geospatial is. It's an integrated 3D mapping stack involving the front-end and mapping components. I'll be showing a brief overview of it in a minute,
04:42
just to show you guys where VDS MapRoxy stands within that system. It's an HTTP 11 server, which facilitates powerful web acceleration and yet performs no application-level caching. And I've put smart decision in brackets,
05:03
because I actually believe that this three combine to a very strong design feature of MapRoxy. It just makes use of the existing standards, smart equivalent of app acceleration standards, and it does not attempt, unlike some other systems,
05:20
it does not attempt to replace them through application-level caching, which in the end usually tends inferior to what's already well functioning and just needs to be leveraged. It's a Unix-native demo, which is written in C++. That's perhaps an interesting observation. Today, even in geospatial, most server-side components are based on scripting languages.
05:40
So MapRoxy kind of, VDS MapRoxy kind of goes along the codes against the stream in this respect. And it's mostly operated through a command line interface. So it's going to appeal to people who are kind of used to the Unix philosophy. And it's configured mostly through JSON files. So it's a fairly easy thing to work with.
06:01
Some interesting trivia. It currently has, I checked this morning, it currently has over 33,000 lines of C++ code, which means it's like a modernized project, but actually quite lean, considering what it does. It's three headers, 88 models, just for you to get an idea. It's actually not so difficult to get a grasp of once you spend some time with it.
06:22
To give you a bit of a historical overview, we started VDS MapRoxy in 2015 with the rest of VDS geospatial. In 2016, it was already a fully mature system, which was massively operationally deployed. It was providing the current surface interface for dynamic TI and generation,
06:40
the TMS interface for on-the-fly coordinate system conversion for massive raster datasets. And by meaning massive operational deployment, what I mean actually was that it's been used to run a website which is attracting something like half a million users per day.
07:04
In 2018, we added a command line utility called MapRoxy setup resource, which made, let's say, MapRoxy accessible to the masses, because resource configuration in MapRoxy used to be very difficult. Now, thanks to this tool, it's very easy. And in 2019, and specifically for the FOS4G conference,
07:22
now this one, the one in San Diego, but let's say for this FOS4G season, we added the WMDS and Cesium terrain provisioning. I promised to tell you guys where MapRoxy stands in the VTS geospatial stack.
07:40
It's just for curiosity, I don't really need to know this. It's basically VTS geospatial involves numerous components. Oh, whoa, whoa, whoa, sorry. Wrong button. Okay. So it involves many components. The rendering libraries, the backend servers, the command line utilities, data sources,
08:02
and MapRoxy is one of the backend servers within that ecosystem. Now, I'm a bit of a proxy myself here, because the actual author who wrote the component is a guy named Vatsalp Vojic, one of the nicest and also smartest people in the C++ development universe, also an extremely shy guy.
08:20
You are never likely to see him, but he is very happy to respond to your emails and to your issues on GitHub and all that. Now, if you want to use VTS MapRoxy, you need to get at least a basic grasp on some of the VTS MapRoxy concepts, so it's going to get a little bit dense and boring, but please bear with me for a moment.
08:43
Basically, the basic concept is the data set, the data to which the proxy access is provided. It's a proxy, so there's got to be something behind it. It's the data set. And there's the interface. The interface is basically the network service or format which is provided at the top of the particular data set.
09:02
So there's two formats involved, one with the data set and the other is the format of the interface. The driver is like an internal MapRoxy functionality which can provide an interface to the data set. That's what connects interfaces to data sets. And once you connect interfaces to data sets, you get something that's called a resource,
09:21
which is a data set configured for a specific driver and thus available using one or more interfaces. So there's a theory behind it. And one more piece of theory, data set formats, they can be anything GDAL or GR can read.
09:40
MapRoxy massively relies on GDAL functionality, so whatever you can read with GDAL, your MapRoxy can work with. And also there's open vector tile support in MapRoxy, also known as Mapbox vector tile. We actually wrote at MelonTech a GDAL driver for Mapbox vector tiles. I don't know why nobody did it before us, but that's what we needed to do for this project.
10:04
The interface types that you're going to encounter in VDS MapRoxy, there's two raster types. One is TMS, which is like a generic tiled map service, named somehow confusingly, it's not the infamous TMS-OS-GEO standard. It's just a very basic approach to the,
10:22
like an XYZ raster server, that's what I call it. And there is the WMDS, which is the actual OGC specification. There is two 3D mesh interface types, which is service, which is specific to VDS geospatial,
10:40
it's a 3D mesh, which VDS geospatial renders. And there's terrain, the cesium-js-terrain provider. And there's one vector format, which is geodata, vector geodata, which is specific to VDS geospatial. Five interface types within MapRoxy. Now, one last piece of theory, the reference frame.
11:01
Well, reference frames are essential to VDS geospatial. It's basically a key concept, it's a blueprint for data interpretation. Reference frame defines spatial division, or the way in which tile hierarchies are organized. The reference frames in VDS geospatial are pre-declarative, which means that
11:20
to present VDS geospatial, you can define as many tile hierarchies as you want to. But luckily, VDS backend comes with many reference frames, pre-defined, and you're always likely to use one. So just for some sort of an idea, this is the most commonly used reference frame on earth, which is Coleman 1-2015.
11:41
That's actually based on that Mercator with CMOS polar cap handling. For cesium, we needed another reference frame, which is TMS Global Geodetic. And for extraterrestrial projects, like the Mars ISO, we are using these quadrilateral spherical key-based reference frames,
12:02
which are good for planetary modeling. Anyway, long story short, the choice of a reference frame is part of resource configuration. And you are extremely unlikely to use anything but these two reference frames, Malone-2015 or TMS Global Geodetic.
12:21
And TMS Global Geodetic, you'll be using only if you're into cesium. So basically, if you're on earth, most of us are with our projects, then Malone-2015 is the only one you'll ever need. And that's the only thing you need to know about reference frames in VDS. Now, how do we actually use VDS Map Proxy?
12:46
There's diverse deployment scenarios. One of them is that you use it as WMTS server that just works perfectly for you. In this respect, you can fire up your QGIS and use all your resources in VDS Map Proxy.
13:01
If you ask why do you actually need to do this, when QGIS already has pretty good handling of large data sets? Well, the reason is that if you do this in VDS Map Proxy, then you get all this functionality in a single place on your network. And it's going to be fully accelerated using HTTP 11 or two for repeated use. So you're going to be basically leveraging
13:20
that caching power that QGIS has implemented for all your users within your ecosystem. And another option is that you're going to use it as an XYZ tile server, I don't know, for leaflet or something like that. The other option is that you're going to use VDS Map Proxy as a CZMJS terrain provider.
13:42
And you can, of course, use it as a service bound layer and show it to the provider who is in VDS geospatial. If you're using VDS geospatial, that's what Map Proxy was actually created for in the first place. And now, all these roles may and actually should be performed by the same server,
14:02
because interfaces to the same data set accelerate one another. That's the way how Map Proxy is built. So if you actually have all this within a single server installation, that's all good, and that's actually a smart way to go about it. This is how you install VDS Map Proxy.
14:22
If you got, or you send Ubuntu server, you could probably outbuild for other Linux distros, but this is, it's as simple as this. Now, I did a brief example of Map Proxy usage here, and that's, I've been working with two data sets
14:41
for this talk, and the two data sets actually come from a single DEM. It's the USGS three second, three arc second DEM in northwestern United States. And I've created two tips from it, like one is the actual DEM,
15:02
and the other is the hillshade. And so you can just download them from my own technology's CDN network. And after you do this, what you do with them is that you run the Map Proxy setup resource utility
15:22
from command line. You basically run the command line utility twice. We're going to be, for two reference frames, Malone 2015, so that we can get VTS geospatial service and TMS global geodetic, so that we can get C0, basically. So this is what we do for the imagery-based resources.
15:43
And afterwards, we do pretty much the same for DEM-based resources in this example. And what we get from this is, oh, we do one more step. That one more step is, you don't really need to go too deep into this, but the thing is
16:02
that we'll be taking a look at the introspection API in VTS Map Proxy, and basically to have a nice introspection web application, you need to bind the imagery resources to the terrain resources. So this is done with these two lines entered into the resource configuration files.
16:21
And we're pretty much done at the moment. And what we get is a browsable directory of all interfaces which are currently provided by Map Proxy, and this is what the resource template looks like. So we just point our browser to local host, and go to Map Proxy, and fill in
16:42
the appropriate reference frame, the interface that we're using, and the two IDs that we used when running Map Proxy setup resource. And I actually got the, yeah. So before we look at the introspection, there's five embedded resource reviewers. For the TMS service, you get leaflet.
17:00
For VMTS, you get open. For WMTS, you get open layers. For Surface, you get VTS Browser JS. For terrain, you get Cesium JS. And for your data, you get VTS Browser JS. So, I can do the live demo now. It's going to be difficult without the mic, though.
17:27
So, this is actually running off my computer. This is the browsable directory I was telling you about. Yeah, well, you know. It's a little off the culture, yeah. Yeah, it was the, so basically,
17:41
this is the leaflet viewer for the TMS resource we just defined. What you're looking at is actual leaflet viewer of the resource which is being generated live on the fly as I browse through it on my computer.
18:01
This is the VMTS interface, and which is using open layers, the very same resource. So I could just use this, the WMTS capabilities file and fire it up in QGIS. And this is the, all these pages are my proxy generated
18:21
and are part of interface if you look at the URL. This is Cesium, which is the embedded viewer for the terrain interface. And in the last step, I got VTS Browser JS,
18:43
which is slightly better performing on the same data. It's like a general phenomenon. So, even though this actually looks similar, the underlying framework is very different. We are using very different tile division scheme and there's like zero overlap in the technology which is being used for rendering.
19:04
So you're free to replicate this example. It really takes just a few command line tools and a single file that you need to edit if you want to play with my proxy. That's pretty much it for me. And if you want to learn more about VDS, my proxy,
19:21
then you can just check out VDS documentation at vdsgeospatial.org. Also, the GitHub repository is worth visiting and from time to time, interesting information may be found also on the Mellon Tech account. So that's it for me, thank you.
19:44
Okay, thank you, Andrei. So are there, we have time for questions, so feel free to ask. Yep, over there. Can you talk more about the caching?
20:01
That was that, or was or was not happening? Oh yeah, I was kind of making the point. The question was if I can talk more about the caching and basically why we don't do it. Well, you might have noticed that there's actually lots of caching happening, but none of that is actually part of VDS Map proxy because the way network streaming relies on TCP IP
20:24
and in TCP IP you got lots of application level caching standards which are actually very cleverly designed like HTTP2 or even HTTP1 caching mechanisms are done extremely smartly and whenever you do like an application level system, this one side, the other side you got file system level caching
20:41
which is done very smartly as well. Like I could probably make a better point for application level caching which is going to supplement file system level, but on the network level, application level caching actually makes no sense at all. It's such a smartly designed system, you just need to know the technology, you need to know HTTP and you need to work with your headers properly and chances are
21:03
that you're going to get a caching mechanism which is going to be a lot more efficient, distributed and working across the entire internet than any kind of application level caching that you might be possibly doing in your application. At least that's the philosophy. When we designed VDS Map proxy, the guys who wrote it and designed it
21:22
like myself and Vasik, actually we were originally network streaming people, not geospatial people. So we've been designing network streaming services outside the geospatial domain for decades. So it was a very natural approach to actually leverage
21:41
all the caching capabilities within the CPIP design and not to do application level design. And I think it's a sound philosophy, opinions might differ, but just try it out. We will see how good a job we have done with it. Okay, any more questions?
22:11
If you see someone behind. Cool, okay then. Thank you Andrei very much and we can move on.