Leaflet + UtfGrids + d3.js = liquid fast, massively scalable interactive web map & data visualization

Video in TIB AV-Portal: Leaflet + UtfGrids + d3.js = liquid fast, massively scalable interactive web map & data visualization

Formal Metadata

Leaflet + UtfGrids + d3.js = liquid fast, massively scalable interactive web map & data visualization
Title of Series
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Open Source Geospatial Foundation (OSGeo)
Production Year
Production Place
Portland, Oregon, United States of America

Content Metadata

Subject Area
I will discuss and demo how I use Leaflet, UtfGrids, and D3.js in concert to view and interact with large geographic data on the web. This presentation will not be on d3.js, but rather how to get geographic data from a map to a d3.js chart. I will illustrate why this stack is liquid fast and massively scalable and discus in some detail what a UtfGrid is, how it works and how to create and server them to the web. The context of my work:I am currently working on an open source project called OpenQuake. As a part of this project we are developing a platform which serves as a hub for integrated risk assessment. It allows users to combine seismic hazard, risk and social vulnerability in many different ways in order to obtain output for science, risk assessment, risk awareness and risk management.All my work is available on Github and links will be provided to all demonstrated material.
Keywords Leaflet d3.js utfGrid tilemill data visualization liquid fast
Web 2.0 Area Arm Hazard (2005 film) Demo (music) Interface (computing) Projective plane Hyperlink Bit Endliche Modelltheorie Computing platform Vulnerability (computing)
Point (geometry) Web 2.0 Curve Hazard (2005 film) Mapping Personal digital assistant
State of matter Code Outlier Multiplication sign View (database) Zoom lens Set (mathematics) Parameter (computer programming) Web 2.0 Bit rate Semiconductor memory Cuboid Area Curve Mapping Gradient Electronic mailing list Bit Category of being Arithmetic mean Uniformer Raum Volumenvisualisierung Website Resultant Point (geometry) Functional (mathematics) Observational study Control flow Streaming media Event horizon Metadata Attribute grammar Frequency Population density Energy level Plug-in (computing) Scaling (geometry) Demo (music) Key (cryptography) Hazard (2005 film) Tesselation Polygon Database Line (geometry) Cartesian coordinate system System call Personal digital assistant Query language Point cloud Spectrum (functional analysis)
Point (geometry) Computer file Open source Multiplication sign Range (statistics) Zoom lens 1 (number) Streaming media Client (computing) Shape (magazine) Open set Mereology Theory Metadata Web 2.0 Preprocessor Frequency Mathematics Goodness of fit Computer configuration Core dump Energy level Cuboid Representation (politics) Plug-in (computing) Intelligent Network Electric generator Information Hazard (2005 film) Mapping Key (cryptography) Tesselation Polygon Interactive television Electronic mailing list Bit Hecke operator Database Cartesian coordinate system Entire function Subject indexing Computer animation Vector space Volumenvisualisierung Right angle Library (computing)
Point (geometry) Pixel Identifiability Multiplication sign Plotter Web browser Code Event horizon Metadata Medical imaging Lecture/Conference Semiconductor memory Social class Vulnerability (computing) Area Time zone Curve Graph (mathematics) Arm Information Demo (music) Key (cryptography) Tesselation Gradient Projective plane Interactive television Bit Price index Type theory Computer animation Mixed reality Website Quicksort
Point (geometry) User interface Weight Bit Price index Mass Cartesian coordinate system Distance System call Number Data management Arithmetic mean Quicksort Vulnerability (computing)
Point (geometry) Bit Tunis
Demo (music) Multiplication sign Energy level Bit Line (geometry)
and we use and I work with the genome Foundation the gym Foundation is uh is supporting a project called open quake and that over quick is it's a platform for seismic risk hazard and social vulnerability and so on so on what we're working on is were modeling Our exactly at seismic risk hazard social vulnerability and building a platform that integrates on datasets relating to those of models that that the scientists are able to produce enough platform on the platform also as an interface for calculating arm and calculating models based on seismic risk and and social vulnerability on the the the web links can check up the projects and learn a little bit more about the project on so basically the stack so so the stack of that I wanna talk about today includes leaflet D 3 and U of grids and the reason why I am using this stack is to tackle problems of big Data noninner put quotes were around that because everybody has their own definition of Big Data on but I wanna bring a lot of data to the web and I want that I want that dated to be interactive so I needed I needed some tools where I could could interact with the data end but it all needed to be brought to the web and so on here's just a quick demo before we get started we're going to just take a look that's on but some of these tools planning work the area so
basically I'm going to explain throughout the rest of the talk what's going on here on and so on and just walk you through what's happening here is but some points have been have been rendered to the web and each point on this map represents has the curve or actually in this case for a hazard curves so I might have a million hazard curves that I need to bring to the Web each each point has on 4 curves each curve has 20 points plus minus so for me this this was this was a trick this
was something I had to had to figure out how to solve this from here's another 1 I made again
you click on the map in this case was a polygon and you get a d 3 chart and so on and the 93 charge you can interact with in this case I've got this on breaking point so the user can set the breaking point and then and then they can identify are outliers of this data so so in the in the grade you see the mean and the red is something that's that's not is breaking point that the user can set and here's
another really simple 1 it's just counties in the US you can you can just clicker hover the mouse and it renders the sketch of the aspect ratio pretty terrible but on it renders this this decree chart so on so that's kind of the demo that's what that's what I want what I wanna talk about on yet so what is going on there what's happening so you grid is being uploaded into into leaflet and then there's leaflet plugin that's just wonderful called UTF grid and that you to good plugin allows I forget Jason and on it allows to get get this Jason array out of that you T of grid and then I take it that way and applied it into a d 3 chart so a few things to note on I mentioned that I might have a million hazard curves but I'm not rendering I'm not bringing a million points to the web at any single time so on what what ends up happening is when I'm zoomed way way out then the great it is is not dense at all and as a zoom in I get more data has a zoom in the grid becomes denser and denser so basically let let's just quickly talk about what you TF grid is on you can look up of the the you are you the UTF spec UTF desk 8 on Wikipedia it'll described that spec what that is on in this case it's a of grid and so that is basically an SQI light database that that's the form that it's held in mom but what it is it's it's it's it's a it's an encoded not when each of these encoded characters that corresponds to a Jason array and so each feature is referenced by the same character and associated to the G some data this is all from box on the website of the result is a cheap and fast fast lookup even in an annex box 4 4 7 PM can handle on so again miniatures describing what's what's happening here on so when you some when you get you're you're bounding box and the pan or zoom of it's going to bring in these tiles so the tiles the tiled UTF grids and it brings in a set let's 8 6 for the area that you're looking at and as you zoom in you get more and more and more of these things on but you only get the 6 of the time or 80 however many it takes 2 to fill that bounding box events in level on once that once that you t of grid is loaded there's no more calls to the back and all of the Jason is in memory at that time and so then when you get that click event for that hover it's instantaneous the expense was when you panda when using and that's when it does that that's when it does AJAX call so does this scale and yes it really does so it's a it's period it's a pyramid tile set on that you can either precompiled or you can serve on on the fly and on but see so so it's working on that on that that that the density of the data that's returned that's that's the magic on just another note so basically I mentioned that it's sq like database so what I like to do is I like to integrate metadata into the SQL database and then I had my application can do a look up on the metadata and understand what the state is and what it should do with it I'll talk about that a little bit more in a little bit on so what you need to get started with this you basically need leaflet you need this plugin that I mentioned before you chiff created you need d 3 if you wanna render that data to the 3 or you can do whatever you like with it and then you need to serve the the UTF creates to the to the web and to do that on my views tile stream Server tile stash or you can use the cloud and you can use a map box account so I mean I'm just going to dive into some some code I mean I don't know how our interests of everyone is in actual code it would look at what's going on but here it is of get Jason that will return the metadata so called in this case I'm calling the tile Stream API and I'm getting in some some attributes that put in and I I get I forget the name the category the tight Our whatever else and this like I said before I got a leaflet application running query and then I can tell that application OK the user has loaded this UTF grid what is it to tell me about it is it is it has the curves is it is it lost curves at uniform spectral curves whatever have a whole list of these things that interact with so once it figures out what it is that it renders it and then on basically I just create a variable wanted to say OK this is my duty of grid and would slow that thing in so here's the here's my tile stream URL and there's the selected layer the 1 of the clicked on and off and then I can just add that to the to the map and OK so then it's in the map it's loaded now the user comes along and they click on that thing or they hover and our and then what happens so now you of grid the duty of grid leaflet plugin on that click event it is it grabs the encoded key and it looks up the Jason array and Jason array can be anything anything I put in there it's just adjacent rate and it's up to me to to do whatever I wanna do with that user so in this case i'm just gonna say OK on let's take that Jason array and John let's fire up this is exactly the same functions was before it's over the last line and that last line fires off my 93 chart and it passes the probability what's that should be problem but probably passes the probability the latter the latitude longitude in this case that's all around and that's all I need for the D 3 chart and then it's just that the future from there on out the future accepts those parameters and there goes the chart so so that the study is on G 3 and how to use d 3 this is all about getting data to the 3 uh from some amount of so a little bit more about serving UTF of creates on so you the these
options is the ones that I would recommend tile stash but so basically tile stream server it pretty compiles the entire period upfront and then you get that as as a UT of grid tau form and that's the SQ light database that I referred to earlier you get that whole thing and then it's very quick whenever 1 any when anyone looks anywhere it retrieves it on 4 and it gives you those tiles that are needed title stashed can do that but it's better generating the tiles on demand and then you don't have to precompile all that stuff so if you've got from our zoom levels 0 all the way up to 22 for the whole world it this can be costly that the the tell stream of of precompiling it's very expensive on once it's done it's all done but then you can go back and change things so it's nice with tiles stash you can set you can point that tile dasher oppose post URL database you can change that data and then you can erase the cash and then when the user are requests tile it's can be updated but on it's slower because as generate attach every theory has to generate the tile every time except when 1 is 1 1 is 1 1 is demanded it's been cached that's the idea on so then it becomes faster faster you can also use not box and there some other options as well as to who's using this stuff on boxes using this and what are they doing with it the the rendering to the web the entire oak OpenStreetMaps dataset so that's a lot of information of the range the web Carter DB uses this technology map Nick is the core technology that's underlying the tiles and the creation of the tiles on tips when creating and the tiles on shapefiles are can be made very fast with this shape file indexing CV not box you need map make installed and then you can run this nifty where's the shape index and then your shape file pre it creates an index file on there and then when you when you when you precompiling those tiles it's much much faster and using our tiles tile middle to on to create those but to create those files out to do the precompiling on you can use as to light as your as your source for the for the as a data source it's on slower definitely noticed before but so you don't get that pesky shapefile Colin living and so now I can put a ridiculously long Jason array in there and I do and I love it because then in my applications I can just have so much data don't always know what to do with so on and then and then you can use post SQL as you as you back end and that's that's great for animations before a lot of other reasons to on a back to the metadata and so when I'm using tile stream server I mentioned before I've got this extra like database and I can just do I can just open Q as Q like 3 on and I can do just to insert on and then I can just say OK my Kataria there's my application is and I've got a huge list and I'm actually I'm actually even on adding into the metadata horror of the part of the value that's used in the D feature those in in in my keys with the hazard curves the was of X axis is constant it's always the same it's only the Y axis the changes so I actually put that right in here and then I've got it for the for the entire use of the application the use of that army tell throughout the use of applications on OK so what about vector tiles on now now I'm a little bit diverging from from everything I've said before everything I've said before his been Rasta based all this stuff is Rasta P G's on so now we're timeout vector tiles and this is all kind of new to me on I've only been experimenting with this i haven't built any applications using this technology but I like to talk about unless on the so what the heck are these things and basically it's just a vector representation of those features just 1 tile at a time it's just the same concept only only were time of vectors on how does 1 create a vector tile on again tile stash does it you can do these with not box on 1 of the guys at my parts referred me to this Tessa I've never used to on the and then not how does 1 rendering render and interact with vector tiles on the client now OK that that's great that you can generate vector tiles but what the heck are you going to do with them you can render them into your leaflet act with some plug-ins or the OpenLayers plug ins as well but the problem is now you've got 10 thousand or a million vector features that you're dropping into on into leaf litter players and it's just gonna fall over on so there's this new there's this new thing that box GL it's so JavaScript library which has of some very very interesting features like I said i've i've only played with this and I can only I can only on I'll talk about I don't have any any real concrete experience with it it is very interesting on so why care about vector tiles and and what can we do that so special well the crazy fast and the ridiculously scalable even more so than the Rasta friends on but more interesting than that they they create some interesting possibilities are and 1 of those possibilities is that you can move the styling on to the client side so now all of a sudden you can have back back to those millions of vector points or polygons or whatever you've got you can now start to style those in the client and done and that opens up a whole new parts of the whole new world of possibilities for for solving problems so I cannot blaze through that really fast I don't know how I'm doing on time and good on time so I don't know I don't know what else wanna talk about her I can just opens up to questions or people conscious of course approach me afterward I love geeking out about this stuff I can show more Donald's if he has 1 on subdued eyes so if you use 1 just getting the questions you have yes so I don't think I think a
cut this we wanted to work with check your site so basically your storing sort of the identifier in the key of the grid and then the most address the data you get with another eject collar where where you we now it's become much within the metadata that's preloaded persuade that 1 right right right so on so the the coding is just to understand where that click event has happened but stood Jason he's loaded with a with every pan and every zone the Jason is there it's in memory yeah does that make sense of what we would call it yes that works yes yes but because it's this it's not this in it's not a million points it's it's this it's this granulated I said at the subset because of that you not low you're not over overloading your browser with Jason it's just as it's just a very select subset of of of the Jason yeah it and and so on I'm sorry I have you taken and because I know through tile etc. you can take the metadata and convert that over to you get pretty try to do that with Rasta data so I can classified land image and something like that I of the Yankees I have but the problem is that you Rasta data has only got 1 value per pixel so that's all you're going to get in the in the UTF grade once you click which is going to get that 1 value that that that that the roster has associated with each individual pixel it does but you may have the other information associated with that class so like you know a soil map or something like that would be the fence and so quite often yeah you need other information associated with that class yeah you could do that I haven't got and yes I have experiment at all with our marker rendering and the 3 JS with projections potentially for animation or more alive user markers no I haven't but but I want to I want to get into that and I wanna mix I wanna make some of this stuff with exactly that type of stuff but no I haven't so far it's just like that demo I showed where it's just I click and open up a D 3 chart plot plotted but just a simple chart graph little bit of interactivity on we were really through the that we really fast through that demo I don't know if you saw on when I click when when the click event happens on that created it renders the G chart there's also that interactivity and not just that blew away the assigned as I work with who wanted to visualize this and interact with the stated that they had access to at every point on every curve for every curve out of a million images it really was like it was really a big win yeah it yeah anyone but I don't know how much time we have I could open up another demo when we look at the Dunn was a little bit more you what your major areas a lot and I really fast sorry reactors blew through that I had this time that follow a little bit of arms so here here's the here's
1 of these demos so basically this is social vulnerability indicators on and and uh so what the user can do is they can come and interact with the stated and they can come down here to this d-tree 3 chart that are set up and down what 1 thing that they can do years they can they
can change the weights that are associated to all of these things so you can have a disaster manager on the ground and they can go in and they can sing population is is much more important to me and economy is a little bit less and and I value the weight of education a little bit more so than you update this thing and then you see that the
the points they're representing the their weight and that weight is then used when no when the user then clicks on 1 of these 1 of these districts and renders the charts so now on using this is on I don't I don't really wanna get into all the details of what all these charts mean but on here you can see I have a number of charge for each district and these are a little rough these are still 1 still kind working on these but on the the 1 I'm really happy with the on this this is this primary indicator chart so basically when he passed me this data I was I I can even understand what it was that they wanted to get out of this chart it was just it was just on nets and you can see I'm now on using that you t of great I'm passing quite a bit of data through that you grid into my into my chart but it's just this huge massive indicators primary social vulnerability indicators and their the animal 35 on axis another 20 going and so then 1 thing that they said to me was well on looking for that outliers so that well OK let's stick with it the mean in there and that's in this sort of thicker gray in the background that's the means an idea I has a little bit of something to to look at and then and then I started in assert thing about flying in red those indicators that pass through this call this chart and or a certain distance away from the mean and that's what I'm calling is breaking point and then I can control this breaking point with this little user interface here so website can change at 2 to 3 OK so
now everybody's everybody's just feeling that on but then I can go back to like
7 point maybe 5 and now I can fine tune so that the user can fine tune or how what values are going to be on discriminated against and down and highlighted there so let's let's get this even a little bit on it just get 1 could be
anyways so that's that's the gist of it and then and then it again you can you can get in here and you can hover over these lines and then you can understand what district that was and what what administrative level and I think it's going to take a little bit more messing around to get into any other demos but that's basically just are in time thanks everyone and then please thank you few and like I said before just feel free to approach me talk to me about this stuff is all about just the of