HERE XYZ Deep Dive on Creating Custom Web Maps
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43467 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Bucharest 2019117 / 295
15
20
28
32
37
38
39
40
41
42
43
44
46
48
52
54
57
69
72
75
83
85
87
88
101
103
105
106
108
111
114
119
122
123
126
129
130
131
132
137
139
140
141
142
143
144
147
148
149
155
157
159
163
166
170
171
179
189
191
192
193
194
195
196
197
202
207
212
213
214
215
216
231
235
251
252
263
287
00:00
Local GroupTexture mappingSoftware developerProcess (computing)Common Language InfrastructureMixed realityVapor barrierFormal languageMessage passingOpen sourceProjective planePoint (geometry)Group actionWeb 2.0Connectivity (graph theory)SpacetimeMultiplication signMechanism designBitCycle (graph theory)Content management systemServer (computing)MathematicsData managementMobile WebBlock (periodic table)Client (computing)File formatOffice suiteNormal (geometry)Complex (psychology)Real-time operating systemInteractive televisionVector spaceState of matterFile viewerPlanningINTEGRALPlug-in (computing)ImplementationWordLevel (video gaming)Physical systemInformationWeb browserWeb pageInternet forumSimilarity (geometry)Shared memoryDemo (music)Different (Kate Ryan album)Open setWebsiteStandard deviationNeuroinformatikFunctional (mathematics)PlastikkarteLibrary (computing)WritingCausalityDifferential operatorSoftware development kitSound effectArmWeb applicationData structureGeometryStorage area networkWeightQueue (abstract data type)Link (knot theory)Disk read-and-write headVideoconferencingMobile appCollaborationismReading (process)Coefficient of determinationLecture/Conference
08:47
EmailFunction (mathematics)Open sourceVector spaceSpacetimeGeometryPoint (geometry)Coordinate systemChannel capacityData typeContent (media)AuthorizationObject (grammar)Reading (process)Dependent and independent variablesMean value theoremSoftware developerInformation overloadTerm (mathematics)SpacetimeWordTable (information)Web 2.0Content (media)PolygonPoint (geometry)Line (geometry)Texture mappingMultiplication signSoftware developerGeometryCommon Language InfrastructureTesselationService (economics)Normal (geometry)Type theoryMultiplicationOpen sourceDatabasePhysical systemFunctional (mathematics)Information securityAttribute grammarDemosceneCodeMoment (mathematics)Expert systemCartesian coordinate systemNumberIntegerTwitterSurjective functionBlogCommunications protocolBuildingBitPower (physics)Field (computer science)Interactive televisionPersonal identification numberHexagonProgramming paradigmModal logicAreaFocus (optics)Position operatorLink (knot theory)DataflowWebsiteLevel (video gaming)Mechanism designComputer filePoint cloudData storage deviceIntrusion detection systemWeb applicationSystem callData structurePerspective (visual)PlastikkartePerimeterServer (computing)Nichtlineares GleichungssystemPatch (Unix)Object (grammar)CuboidFront and back endsZoom lensDefault (computer science)Category of beingWeb-DesignerRepresentational state transferSound effectVideoconferencingHecke operatorVideo gameComputer fontGroup actionDisk read-and-write headArmFrame problem1 (number)Lie groupUniform resource locatorWeightCausalityNetwork topologyTask (computing)Archaeological field surveyComputer animation
17:27
Auto mechanicDatabasePoint cloudService (economics)Event horizonHookingCASE <Informatik>Complex (psychology)Goodness of fitMappingServer (computing)MathematicsTexture mappingClient (computing)Connectivity (graph theory)Multiplication signView (database)Computer architecture2 (number)Physical systemFormal languageDatabaseData conversionTelecommunicationProcess (computing)Queue (abstract data type)Message passingConnected spaceBitGroup actionTouchscreenDemo (music)Mechanism designEndliche ModelltheorieSpacetimeMultiplicationSatelliteVideo gameOpen sourceDifferent (Kate Ryan album)GeometryComputer fileProgramming paradigmNeuroinformatikProjective planeTesselationCombinational logicOperating systemMean value theoremProxy serverWeb browserField (computer science)QR codeVector spaceZoom lensPattern languagePersonal area networkPoint (geometry)Data structurePointer (computer programming)CodeUniform resource locatorPlug-in (computing)Web pageEvelyn PinchingSimplicial complexVirtual realityService (economics)Raster graphicsOptical disc drivePoint cloudRevision controlVideoconferencingDatei-ServerDevice driverSoftwareFocus (optics)Scripting languageTime zonePlanningLine (geometry)Scaling (geometry)AdditionTable (information)ArmMoment (mathematics)Raw image formatLevel (video gaming)Lecture/Conference
26:07
CollaborationismPoint (geometry)Open sourceSpacetimeHeat transferRevision controlBitBusiness modelPattern recognitionWeightPhysical systemLevel (video gaming)Presentation of a groupWater vaporFunctional (mathematics)Game controllerDatabaseNeuroinformatikEndliche ModelltheorieComputer fileOrbitRobotCASE <Informatik>MereologyPlanningProjective planePattern languageLimit (category theory)Shared memoryGeometryMultiplication signMyspaceCore dumpGoodness of fitComplete metric spaceReplication (computing)Computer animationLecture/Conference
Transcript: English(auto-generated)
00:08
Okay, then I start I take every minute I can get I try to be really quick But I also want to give you a deep dive so that might get a bit intense We can talk afterwards if it was too quick
00:21
So I'm Oliver. I'm working at here. I'm director for interactive data at here and we're happy to be at first 4g again We've been in San Diego when we launched XYZ and I wanted to tell you a bit more about What XYZ is but first of all, let me quickly give you an update for those of you Haven't heard about here We are a company which is primarily building map data for industrial processes. So essentially doing sensor data
00:48
Having cars out there in 200 countries driving around and updating the map More recently we have been bought by car manufacturers and they have been very Supportive of our journey where we wanted to say we want to become more open source
01:04
We want to become more involved in the community. So our open source journey started 2015 primarily around open source tools because one of the big problems that a big company has is making sure that we Honor all the licenses that are in the various libraries that we're pulling into our tool
01:23
So we build that tool set with a couple of partners like Microsoft and Amazon and others We are very active with the open source review toolkit, but also with standardizations like SPDX It's not geo-related at all. But for open source, it's extremely relevant as of
01:41
Probably two months ago. I'm also on the technical advisory committee for the Urban Computing Foundation, which is a Linux foundation offshoot focusing on Smart cities essentially and a stack and an infrastructure and open source projects Supporting open source projects around Urban Computing It's just getting started if you think there's a project that you would want to bring under that umbrella as well
02:05
Talk to me or go to the Open Computing Foundation website to learn more now to XYZ what we're here to talk about essentially is If I take a look at the state of the web maps Today, it's still a bit like okay. It's mostly read-only you can see maps on the web
02:25
There are few creators of web maps mostly professionals probably mostly also from a group that has been educated in GIS Geoinformatics and who really know the tools that they are using And it's mostly static I mean it's data that you prepare at one point you work a lot with the data and then you publish it you bring
02:45
It on the web you share it with people If I take a step back personally Then it looks to me like it was it looks like the web at 1995 where the web pages were mainly static there were a big couple of big
03:01
contributors to the web But it's not as interactive as we know today yes, and we got vectors and emojis But that is not really making up for a lot So a lot of Technology and innovation has happened on the web Blogs, chat, Flickr, so we are expecting much more interactivity much more real-time turnaround of what's going on and
03:25
a couple of guys in here thought about can we Play the mad scientist and actually think about can we turn read-only maps in really read write maps where you can Collaborate where you can work together without having to install a lot of complex tooling that I
03:43
Like to talk about normal users and the professional GIS users because normal users already struggle with tools like Office 365 or something that is on the web that is actually very easy if you are a technology person that offer Collaboration, but we haven't seen that really in the space of mapping and GIS tools
04:04
And that has also been a big stumbling block in the company internally, so you can't imagine how often we think In our company we have a cool data said I want to show it to somebody What do I need to do to show it to somebody because that might be a manager who doesn't have QGIS installed
04:22
So how do I get a map that conveys the message that I want to convey? to these people So we thought it's there's nothing really out there that fitted the bill there were a lot of components language barriers between Python and JavaScript and
04:41
GDAL and all the stuff somehow gets in the mix but we thought can't we just somehow bundle that up together and make it easier to use and That's really where XYZ came from so it's think about it as a content management system a blogging System for map data on a very high level it goes much deeper
05:01
And I'm going to show you that it's not only that but we wanted to provide that entry point so we are essentially switching from Words that we're publishing on a block to coordinates that we're putting on the map But still want to edit you want to have a similar moderation edit cycle where you can go in and update maps without having to republish them and send your
05:24
Your audience and you link people should be able to go to a map and see the most recent information that you want to Show them possibly even go back in time and to make that happen All these components somehow need to play together and it all hinges around Apis and I'm going to bit going a bit into the API's and
05:44
Depending on how quickly I talk we can do quick demo at the end to actually show you all that really looks like One thing that we wanted to do as I said is from the engineering team we wanted to make it open It's difficult in a large company that has been business focused to go out and say
06:01
We need to take the stuff that we are building and share it with the community It took a lot of convincing but people were open to that because we are also using a lot of open source internally and we felt We need to give back and take what we have and bring it in front of the community and say is that something that? Helps everybody can we drive this forward the other thing that we wanted to do is not
06:25
recreate something that already has a good presence in the community So we don't want to compete with QGIS or we don't want to compete with GDAL because they are very good Solutions already out there. Why should we rebuild them? What we wanted to ensure is really interoperability that you can take any tool and
06:43
plug our functionality into your Kind of kind of work working day how you're working. We didn't want to change the way you're working We just want to make it easier and that means also open standards open data How do we exchange data which is if I have to pick one problem that I see all the time it's
07:05
Access to data is still way too difficult. There are too many formats out there There are different discovery mechanisms And then you are spending a lot of time converting the data into the right projection into the right format wherever whatever you need whatever you want to show and
07:21
Finally we said well if we are going to do that. Let's take a step back and Try to build it so that it becomes a real-time system so that we can change data immediately both on the client side that can be a web browser or mobile device and Push that change to the server and everybody who looks at that map sees exactly that update that turnaround cycle becomes
07:46
really powerful And it's not something that is easily done you can do it But there's a lot of infrastructure involved and not everybody wants to get into the business of managing infrastructure So this is a bit the reasoning behind it and now I need to talk about this guy, and it's not about Postgres
08:04
It's about the elephant in the room because I'm talking about open source I want to get that out of the door the plan is for complete XYZ including the hub studio viewer JS maps or a maps implementation that we have to be on github by the end of the year the work the team is working
08:23
tirelessly on that But we have to go through our open source review process so that we can make sure that we are honoring all the licenses It's not as easy as an individual Developer pushing out a project onto github. It's more complicated for a company We already have the QGIS plugin integration on github
08:42
Which we can talk about a bit more if you are interested on the booth We have a CLI and the documentation is already out there So that's the current state and from a very high level XYZ studio is really a simple web application where you can go in and edit and maintain your maps
09:00
It's really geared towards users who've never installed ArcGIS or QGIS and who don't know how to work with that a simple website that you log on to and Manipulate data or bring data in that you want to work with and it follows at the moment the complete flow from acquiring data manipulating data and then a very final easy step to hit a publish button and you get a link or an
09:27
iframe code snippet that you can take and copy and paste onto your blog or onto Twitter and we're adding more and more publishing capability and more and more Ways of bringing data in at the moment, it's very much focused on GeoJSON because we see it especially for
09:45
Users and also developers when I'm talking about the API's who have never ever experienced anything like WFS who might look on the Android device at an API and say I'm getting I'm getting three Integers back or three floating point numbers back. They tell me it's
10:05
Latitude longitude and altitude what I'm going to do about this if I want to track my location, how do I do that? So essentially we wanted to bridge that gap from web and application developers to into the GIS world. They are not
10:20
GIS experts at the moment But really more powerful behind the scenes is actually when you are moving out of the out of the world of studio Which is a very easy entry point into custom API's and Bringing data on top of a map a web map that is interactive that you can click on and interact with I
10:43
Wouldn't say this is something that we've never seen before we certainly all know that this kind of interactivity exists But rarely is it connected to a data source that is also live where the data feeds might be coming from different Devices maybe from sensor data that is being collected in the field right away and
11:03
Immediately published into their data store and that's where really the power comes from that means you need to do to do a bit more coding But it's very easy essentially because you don't have to worry about the infrastructure, and I don't know about you if you're building Geospatial solutions you probably most of the time don't want to worry about
11:24
Infrastructure and keeping Postgres running and installing all the security patches and making sure GDPR is a big thing if we are hosting it in the cloud if you have customer data in there This is all stuff that you normally need to worry about and we said is that something that we as a big company can take
11:42
Out of the equation and if somebody wants to do that, it's great You can take the stack you can install it you can run it on your own But then you are also maintaining the server and maintaining all these security perimeters if you need to If you don't want to do that we have a freemium offer that is very easy to use you Don't need a credit card. There is no lock in you can take your data out if you want to and
12:04
You can use it already for commercial purposes, and you're getting access to here API is that we have like geocoding and routing So it's really we want to make it easy for people to try it out and to say this is something that that helps to bring to solve my problem that is where we try to focus on and
12:22
Digging one level deeper here is how do I talk to these back end storages? How do I talk to that and we try to abstract away everything that is very specific for for geospatial stuff and try to make it generic and Really simple from an API perspective just a couple of calls and developers might know the crud paradigm create update delete
12:46
This is essentially what we are using it's a simple rest API So it's a protocol that the HTTP protocol that the web talks anyway And then it's as simple as posting is a little snippet describing I want to create a new space as a post request to our service and what you get essentially is a database
13:05
Provision for you or a database table essentially provision for you where you can store your data and that takes Milliseconds so you can just stand up your data immediately And I probably need to say a word about spaces spaces We deliberately didn't choose the term layers because it's so overloaded in that space already
13:25
Spaces what we wanted to say is let's make sure that it's very easy to use so for our spaces They live with geo JSON data so that means you can take points lines polygons multi polygons and put them in a space and
13:41
Normally if you do that with QGIS or with other tools you need to have different layer for these different types And we said normal web developers wouldn't care about it. Why can't I put a polygon and a point into the same? Content database they might have different attribution that is something that we have to deal with so Spaces are essentially global worldwide layers where you don't have to worry about
14:04
What data you are storing in that or how much data you put in that we are using that functionality internally We're currently managing about 9 terabyte of live active data that continuously runs in that system Which is essentially our own database points pure eyes
14:21
Building outlines all the stuff that we're getting and once you have a space the next thing is you can get content into that We give you some lightweight tools like a CLI or the studio where you can just take a file and upload it If you want to do that yourself You're just posting the snippet of geo Jason into the service and we make sure it shows up at the right position
14:40
We also make sure that we can give you dynamic tiling on the fly so you can request you can just push in geo Jason and you can ask for tiles you would get tiles back and we can even Sort of surgically update individual points individual features where you're saying I don't know that complete structure But I know the name of this football stadium has changed from Anfield to something else
15:04
And you're just posting that update and we are making sure that we can merge all that stuff into the database And eventually, of course, there's a call to get it out again either by ID which is a different discussion how IDs get created But you can also ask for it by bounding box
15:23
Give me everything that is in this bounding box or everything that is in this bounding box. It has a certain Property or has a value greater than or less than anything So the usual kind of interaction requests and it becomes more interesting in the next level when
15:40
Obviously, this is very helpful for low zoom deep zoom levels when you're deeply zoomed in and you want to see the details once you're Zooming out you want to get a higher level overview like a heat map or something We are supporting hex pins on the CLI You can essentially take your content and say Generate an overview of that content so that I can quickly zoom around and work with that
16:03
That is a big focus area at the moment So deep dive what's behind it? First of all for developers. It's one API Out of necessity we had a lot of legacy systems. They were talking very different API is depending on which
16:22
Layer in time they were created somewhere. We're talking SQL others were like XML requests and all of that and every developer had to had to learn a new Interaction paradigm depending on what data you wanted to work with is that let's encapsulate all of that put one API in front Treat everything that is stored in the database the same and then it doesn't really matter where you
16:46
Get your data from you Just talk to a space and that space might contain building outlines or it might contain a GPS trace or some live positions But the mechanics is always the same and you can share spaces you by definition or by by default When you're going in your spaces are private nobody can look into that
17:03
We can't look can't look into that you need to explicitly share that if you're publishing to the world Obviously we need to be able to show it to the world so we need to be able to read your space But we also have a very fine-grained granular Permission system behind it where you can say I want to allow
17:21
Contributions in my space But I don't want to allow deletes on my space or maybe somebody is only allowed to change objects But not create new objects all of these mechanism mechanisms work and if you know a bit about a structure how Computer operating system works you probably eventually see a pattern there that it looks very much like a Linux system
17:43
Especially if we're going to the next view because behind that API Are hidden different mechanisms of accessing the data so on a computer to take that paradigm you have different device drivers that allow you to talk to
18:01
An USB stick a network drive a CD drive a DVD drive But as a user you don't care really how the mechanics work of working with that system You just want to get your files from there and we're doing the same thing with geo data So when you create a new space it lands in our XYZ cloud if you want to But there's also a way through these connectors to hook up different sources of data
18:24
So you could say I have a life API from another system. That isn't even geospatial Maybe we need to do some processing or I have an existing WFS service and you can hook up a connector that does that bit of conversion that is involved for our system to talk to
18:41
Another system that also allows us to connect securely other databases if you say we don't I don't want to give you my data because I want to keep that private But I still want to work with your system Maybe I don't want to completely host it myself Then we can build a connector and hook it into the system saying this connector does a conversion from your private database
19:03
into the language that we need to Communicate to request and update data and this can be as easy as complex as you want to most of the time You're just reading data updating data becomes more interesting But it's a good way to migrate away from legacy systems as well. So you can easily abstract that out
19:23
behind that just five seconds on that Architecture view we have essentially hooked up a UI the front-end side of a JavaScript Maps Component to the back end so that it knows quite a lot about the back end so that you then don't have to work about
19:41
How do I read data from the back end and push it onto the map and when I'm panning and zooming and scrolling? Something needs to be pushed off the map and reloaded all of these mechanics are baked into the API already You just say I want to see a map and it should have satellite imagery and my space on top of that or multiple Spaces on top of that in addition. It has a rich component model behind it
20:03
So we are firing events both on the client and on the server and the server is obviously a bit more complex That's why it's also taking us a bit more time to open-source it because it does not only deliver your data You can also get map tile data mvt data for both here and for the first time for here also OSM data through the
20:24
Maps and project the tiles and project we are collaborating with them Actually, we I have a couple of maps and folks on my team and it's been extremely fruitful, but it goes much deeper Essentially, that's the high-level architecture that makes that happen But it goes
20:41
Conceptually, even a couple of steps deeper where you don't only store your data. You can logically manipulate your data You can build up what we are calling virtual spaces Which essentially are combinations of different sources where depending on how you configure these virtual spaces Something might overlap or overwrite something that exists in an underlying space and this allows us to build for example
21:05
The rich editing capabilities that we have in map creator which is the source for all of that goodness in the API because we built that eight years ago and Battle tested it essentially in the most complex use case we could find in our company You can combine layers together so that you don't have to make five requests to get five layers
21:24
we're just merging them on the fly and It even goes so far it's not publicly live yet that you can hook up into events on the server side So you could for example every time no matter through which client your data changes You can trigger an event on the server side push it in a message queue and do some custom processing on it
21:45
so it's a very powerful and rich communication mechanism to do that and Since I'm probably hitting almost out of time. We can do two things at once One is a demo and at the same time while we're doing the demo we can Have questions and you can watch what's happening on the screen
22:02
So let's quickly see that in action if you have a mobile phone and hopefully a bit of Connection scan that QR code or enter the bitly code and I'm going to do the same thing here on the device Hopefully pops up on the right screen on the web browser and let me click that and hopefully something opens and
22:36
Here we are I still miss my mouse pointer eventually it will pop up on the screen
22:41
I don't know if it's left left right or oops So what we have here is a map that essentially says as a background I want to have OSM data as vector tiles. It might be here data I'm not quite sure looking at that It could be one of those and if you are seeing the same thing on your phone
23:01
You can just tap on that and a point shows up not very exciting at the moment But let me zoom out a bit now a mouse would be good
23:55
These are these are just points on the map but behind that it is you can look at the webpage and it's really super super basic
24:05
We're pushing the tabs that you're making into the space and Everybody else sees the same thing that is happening in this space and even more We have with the QGIS plug-in for example the capability to hook up that space. Oh, perfect
24:22
Is there one space left? Oh, that is super. Thank you
24:46
And now I mean you can tap on the location where you came from if you're zooming out pinch zoom on the phone Oh now it wants to install Logitech software. We don't want that So this is a very simple demo
25:01
But it shows also it's very easy to develop a solution that works locally, but sometimes it just works Well people want to use it worldwide and suddenly you're running into an infrastructure thing where you're saying Oh, how do I scale that up so that it does work in Europe in the US in Asia? How does it work? Partly offline if the client is not connected for example
25:22
Let me go to the world map I'm really happy to hear that so no questions It's actually pretty fun, and we try to capture that and make a video out of it. Questions?
25:43
In the next version we will yes that you can proxy raster data through actually any binary data So essentially what we want to capture is yes, you want to have satellite imagery in the background There are other use cases where you want to have height fields or 3d models that are very specific that you can proxy through that
26:02
System and still have the same mechanism of working with that yes Yeah And I can repeat the questions if you don't want to pass it through So now look super impressive regarding the open source versus commercial part like I Understand that you are moving toward open source Is it also something that make import export possible like vendor lock-in is not only about the license. It's also about like
26:27
How easy it is to move to and out eventually exactly yeah? It's already there So you can immediately there is in studio already a download button where you can get all the data that you put in studio
26:42
You can use the API to pull it out or the CLI We're just saying dump myspace into a file, and you get all the geo JSON data We are also thinking about it to make read-only copies of the post GIS database that we're using Available so essentially a replica where we're saying if somebody wants to use post GIS functions directly
27:04
Why should we not give you access to do that or at least a way to? Replicate from our Postgres into your system because essentially it's geo JSON data. Yes. We want to make that really easy
27:24
Go ahead One way is to get a bit more recognition in the community because last year when we talked to people they were like here What you are doing what? Now we are getting a bit more known, but it's also the business model behind it is that?
27:41
The hosted version is free up to five gigabyte of data Which is already a lot of data and two and a half gigabyte of transfer and especially for the commercial use cases We see if people are putting in a lot of data and use a system and are successful We are saying okay We want to get a share for that and it's primarily what we find already The customers who have existing systems who are willing to switch over and say it is much cheaper to pay the fee
28:06
Than to run the complete infrastructure ourselves, but we want to keep it extremely open actually from the engineering team We've been trying to push and say can we do it completely open and no limits for open source projects? Which is also kind of a pattern we see often in open source projects
28:21
But they were saying essentially we need to cap it at one point because otherwise it would not be commercially Planable, it's not I'm not saying it's not viable Exactly exactly We want to we want to develop it further and also once we go on GitHub and of this year
28:42
We want to work more with the community to actually find out what are good use cases. What are your problems? Can we solve that by using that infrastructure and particularly for me? I try to work with the Urban Computing Foundation to say is that something? that we can put under the governance model of the Urban Computing Foundation out of the control of here and
29:01
Get more partners on board and get more Collaboration on that level good and we're almost over time