CartoDB Basemaps: a tale of data, tiles, and dark matter sandwiches

Video thumbnail (Frame 0) Video thumbnail (Frame 5553) Video thumbnail (Frame 17193) Video thumbnail (Frame 17931) Video thumbnail (Frame 19170) Video thumbnail (Frame 19633) Video thumbnail (Frame 26366) Video thumbnail (Frame 34775) Video thumbnail (Frame 35519)
Video in TIB AV-Portal: CartoDB Basemaps: a tale of data, tiles, and dark matter sandwiches

Formal Metadata

CartoDB Basemaps: a tale of data, tiles, and dark matter sandwiches
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date
Production Year
Production Place
Seoul, South Korea

Content Metadata

Subject Area
CartoDB is an open souce tool and SaaS platform that allows users to make beautiful maps quickly and easily from their own data. To complement our users needs, we launched last year our free-to-use open source OSM based basemaps Positron and Dark Matter (, designed in collaboration with Stamen to complement data visualization. While architecturing them, we had several compromises in mind: they had to be powered by our existing infrastructure (powered by Mapnik and PostGIS at its core), they had to be scalable, cacheable but frequently updated, customizable, match with data overlays, and, last but not least, they had to be beautiful. This talk is the tale of the development process and tools we used, how we implemented and deployed them and the technology challenges that arose during the process of adapting a dynamic mapping infrastructure as CartoDB to the data scale of OSM, including styling, caching, and scalability, and how (we think) we achieved most of those. I will also talk about the future improvements that we are exploring about mixing the combination of basemap rendering with data from other sources, and how you can replicate and tweak those maps on your own infrastructure.
Polar coordinate system Wage labour Divisor Multiplication sign Execution unit Set (mathematics) Insertion loss Theory Cross-correlation Scalar field Cuboid Codierung <Programmierung> Endliche Modelltheorie Associative property Computing platform Systems engineering Boss Corporation Focus (optics) Texture mapping Mapping Channel capacity Information Software developer Forcing (mathematics) Electronic mailing list Staff (military) Volume (thermodynamics) Demoscene Category of being Computer animation Software Personal digital assistant
Serial port State of matter Multiplication sign Sheaf (mathematics) 1 (number) Set (mathematics) Different (Kate Ryan album) Negative number Endliche Modelltheorie Extension (kinesiology) Physical system Boss Corporation Texture mapping Mapping File format Software developer Fitness function Sound effect Bit Staff (military) Demoscene Category of being Process (computing) Phase transition Order (biology) Artistic rendering Website Text editor Spacetime Geometry Point (geometry) Server (computing) Existence Open source Computer file Online help Rule of inference Field (computer science) Number Energy level Associative property Mathematical optimization Default (computer science) Focus (optics) Scaling (geometry) Prisoner's dilemma Database Limit (category theory) Affine space Particle system Word Maize Computer animation Personal digital assistant Statement (computer science) Speech synthesis Musical ensemble Table (information)
Mathematics Computer animation State of matter Associative property Window
Word Mathematics Texture mapping Computer animation Personal digital assistant Hierarchy Planning Mass Variable (mathematics)
Message passing Computer animation Mapping Projective plane Form (programming) Element (mathematics)
Complex (psychology) Dynamical system Open source Multiplication sign Workstation <Musikinstrument> Parameter (computer programming) Mixture model Mathematics Mechanism design Natural number Program slicing Videoconferencing Associative property Stylus (computing) Texture mapping Mapping Information Bit Cache (computing) Uniform resource locator Computer animation Data acquisition Auditory masking Telecommunication Phase transition Website Iteration Quicksort Spacetime
Point (geometry) Pixel Server (computing) Service (economics) Divisor Computer file Observational study Open source Multiplication sign Workstation <Musikinstrument> Set (mathematics) Mereology Mixture model Mathematics Latent heat Mechanism design Roundness (object) Hooking Operator (mathematics) Manifold Reduction of order Electronic visual display Software testing Endliche Modelltheorie Associative property Position operator Mathematical optimization Physical system Area Default (computer science) Dialect Texture mapping Mapping File format Maxima and minima Multilateration Datei-Server Flow separation Type theory Word Computer animation Integrated development environment Software Query language Personal digital assistant Right angle Quicksort
Computer animation
at this this this will few
I'm focus on what the
the sole theory on Hamilton amounting as I'm a systems engineer at Kennedy and I want to give this stuff this this is about the country base maps obtained of the ties and that Madison Wis USA and after all is said and are is obtained like a story of how we ended up at the serving base maps by the just an evening factor of Fargo fond of Sergio it's try to something in the early on Friday evenings have something that we call and than leapfrog it nearly from Friday's which is basically with a about they need to spending some time hacking on top of the stopped for some experiment experimental software think things that we want to improve Auroora's Philippine Martin and this fact and we do this is a not because we had to push around its and it's a way of development and we below their country the start of the new PCs on top of the existing pieces of the categories that for example the decoding is just a scalar functions of on top of the on top all bonfires sky category accounts it would have that all the really they down on news that on their own bosses full-text search capacities to search around for geocoding names and volumes of the data labor datasets are built a which is the which are the inuit if you to carry the accounts on go to create a new association and you level a of how open data that you can use of up out of the box that they actually affected far enough from another different can of in that be account which is 70 there and it's been copied to your own account and so a lot of internal the eyes and things we useful both for development on the scene ovary thing are built on top of the Kennedy because we think is a way to decide improve experience both for us and for and for everyone who wants to give some of all our platform sold by 2 there is not a base not simple and yet complex beside it in a simple is not just layer of data in our case wanted to be some open data and with the matching style that is some models of that is the most difficult thing and so it makes sense for an evening to try to make title create some base maps using Kennedy be even though category you there since it since the beginning it was envisioned as a platform for making maps but for putting a list of information the BOV base maps like correlates of they are often quite a small amount of data compared to a loss and a combined to any other a good that might be worth it to called the base map units information but on top of but most of all stack was already based on both the forces while policy ideas which happens to be like them most common staff also business itself you know we fo cused a lot of and selling dynamic data that changes frequently and it is not as big as the dataset from all ASEM it will believe it could be what this stuff so little and what can and from that we have as you want with the with the in with with a less detailed data set which use natural and the
melancholy and related their things do to make up their base map would just not not in growing level this country's level and try to assign a and we like 3 to 4 base maps using that don't have to be a star with the account and I will upload it isn't all that they all day and that you you I and we will use this to size to how difficult how far did we get on the purpose of being their association in all relate some kind of specialists from a how far went from being a base map data on we were always there I mean you can make a basic phase not using the the the just of loading the data sets and pushing them and making this filing which can get to be retaken difficult to deal with different systems and things but if I add the category editor was not the best suited to for this because of well is out of order you I am you I was in the centre made their to suit such a big amount of layers 1 on top of each other and there was a point where the they overlap in time and it broke and some other things like that there's dataset size mean you get the of 2 D of light thinking about stops but that's word that if we wanted to make a wide dataset we could not imported listen dedicated the you I and that was fine because it usually 0 1 to 2 to upload and display a our 100 geometry but once based not sadly the extension for us that they're not the rule began despite of all these all these particles and the the making of simple example was quite a scene and it was so simple that it could work against a currently the dilemma utterances x y c dies but it does refer it with this the disco through the work of the UCLA ID which is something that depends on both the the editors have changed and the and this style but with any of that because when get to get efforts to us so we would that they put gratuitous of make than make our and Ireland and you next to just negative effect serial your pointing to onto the real long for the association and that was the easy way to have I mean we will really have something that the system is not simply a little without even using Anthony has unique tool that that's the x y format for something that is not a without very much work so it developed like with the 1st phase map and that would with other basement we the the single there's not like a year and a half ago singular the enormity and they were only available Academy of the for long time but what they some almost a year ago wanted to go bit I whether the develop before because with site to wanted to remove them up using it because of if you wanna make social based social amounts and it's not his and by a community and like it into the many to engage people to make maps from being afraid of how many times would we debut it and they wanted them to make them aren't actually want them to be popular who had to remove that this restriction all the maps together the have unlimited mappings but then it we had the day off that is solidly but there was a need something and in the book the book behind and we tried they of course of people a a 7 7 days maps that the very much better than but who instigated try and to me in Macon and space map there that we could ourselves and the here we can that it could be responsible staff and we could see the light of these states of fits and that also was assigned for that association in a sense of the space map is really going to be the default may economy the editor so 100 % up to 90 % of the is it is going to be using to get is from the size is another name on top of it and they let us all relate to want to be as close to data association African so that's why would say across the western limits and you get the help of from the scale to using Boston make definition of makes sense and having the whole SN inside a table which happened to be assigned and the database and account of the Senate from planets with although there was not enough you know it's them that well it makes sense in that it doesn't make sense the way out of 100 and 50 they did not like the always and there with only pose as well for their rendering but we hope that we could like do stuff but some of it and then to cross the medallion it's just this is about 1 because of the name of the basement that matter we we got the help of a statement with help us make 2 and that's awesome in open source maze maps which is positronic model the whites and adopt 1 it would suffer from which it would assign with that of soliciting mind as in but a they're the ones with which are going to be using quality by if also the baby and that's how we got a bunch of interesting stuff and above the irony body to handle the same number so assistant of of the in 1 inside the cavity funding are not our existence this my infrastructure for example they we used a material in this matter is being used to filter the dataset the sections of voice that a we're going to be relevant if it's by the lovely like transferring to many started to many data to tighter server and was there music some uh they're they're very handy and bosses 1 9 4 they can be with refreshed concurrently so it makes sense to use them as a some kind of mural of there was data that we keep updated with both among osmosis with the field so therefore the base maps and also some a lot of SQL magic to make sure that uh the data matches it's speech assume etc. and it was a dumb idea of senescence them and will with and in which focus doing thing then all things ends and then was about to the development process which is a bit was that developed on the Bismarck is entirely over which has got would indeed it also uses status assess it will be much maps for what we want to display but it was fun some issues with the kind assistance handling it is displayed at different with that from on the 1 on the on or it can be modeled the eyes so we decided to just have a book and look for some for something else and this 1 did the initial development was done time it's a we went iterating and another scene on the draft data which is what made on the CML the BJS model ended up a with their creating another whole way to trade base maps and some of them to be which is the optimism of data with this of realism of of the state that is and applying you can you know but interval effect of on top of item which CS
will connect the affinity accounts and allow you to see
anything to see at some kind of CSS and ultimately
will shift to to the and we
can have a just a preview window to display the dataset of showing there and their and the association of creating this state and its school because you can just a change anything for traveling change the watercolor
because words like you tend to think it's so we dealt with a bunch of cool applying for editing and linking and carried editing in and on top of we felt it was like the the right assistance then and DSA Financial it automatically explicit Kennedy
and it generates a new based on with a style you've set you you've sense and you can also not only change the this style which has its own hierarchy and variables in every and any custom processes things you can also choose acquire is that there applied to the to the map it and to its later so you can like for example I just opened a new 1 they up any kind of ways for for example this is likely tumbling out that you can a plane as mass of a
message of form that have the same up dynamic and the projective into another reaction this is Robinson the the it so an element is of
this is that all the base maps are and the use
of our existing infrastructure which is based
in which these focus and dynamic
mapping and making sure that things get up really quickly so you could also mixed on top of the between the layers of that make up you could also make a on a dataset of your own information that you about the things that can be added this KPI of where it we access in Canada you wanna use the and you can use inside of this map it of masks abuse filtering using SQL or any kind of communication 1 achieved as you your prematurely and SQL and CSS and we're just rendering and was a whole the caching mechanisms and humiliation mechanisms in both with the upper bound fallacy and the mixture it will keep the us of the old in almost real-time so we will end up in would Catholics remained in on this base maps and developing new features then again our goal riffraff and none and I go through briefly at the facility societies centers in the sense that they always they follow this much for my site from this couples ever thought that for the association and with within of sources in giving it to things like this this is like a map that is all that just as is doesn't have very much spaces it's covering all layers so when you when it's quite simple it sounds quite simple that it has it a little slice of complexity you can just put the labels on top of solid this intuition and a as the the problem is it was quite a structural layers we just released and differently which was just a bit to the earlier which I have the labels on top so you can just put using the let them come in 1 country DAS but the base not the which is the the thing you see the behind the Blue Mosque and above to put whatever you want a piece wise and the book that you put the labels and it makes finances so changes so as data but this is like a lot of dishonest back in those who have been of a funny issues like while you have to allow the style and it is the labels but it was reasonable to make it but we also add went unimplemented some other things over the kind of the air would send you have explosion and affecting how about this marvelous like because right now that we have the the using the same with labels like faults without telling you had not will be even notice which I think is sort which I think is pretty cool because it didn't phase like natural was implemented some quite things in on time you set out to be able to cope with all of this and 1 of those is videos preview based maps Newton those previously stations and Europe's about a on the maps from entering air the previews actually in the 1st time in the 1st iteration of the the when flat and they got out of the 3 layers yeah but they wanted to go as the father and a meal to serve as they can actually for for 1 for your whole association include the based not on the labels and of course you data I went we went over this by using the Maps API that we really have which is based on we have what which is based on not make and this goal we set analytical assigned which mouth which basically means that you can no longer only request not layers with the stylus CSS inquiries but you as request and as the bilayer it and instability of that allows x y see for example in this this is our home is much but you can put any pretty much any other based movin it and it will request all the latest and compose them together and serve them as another map which is just like a combined mentioned in dB in PNG off that's not in and I think we have it for the for that reason your the confined to request xyz ties that you cannot serve as well so this static not EPI which means that you can request it this silent for a fee assume and center in this coordinates and of a given size and even dissolving the URL of the parameter to see here on the on the and you can just change and take the out you
want you want display of so a the last part is the system spot that time most involving with not because they the base map
uh do indelible obeys not wind up improving our infrastructure below testing and comparing and new settings and it pretty things that we ended up testing with a base maps but expanded to a whole can be the 1st of them is not filing mentoring assessing because it's a simple concept in which is that the the dining services is the 1 subsidized is that when you just for a file server friend investors that title don't it renders the for somebody give your if you want to request this file it will not paying the style but also obtained the whole Addis and a and a bunch of others and dials and if a happen safe so because you know most of the times is going to be people who seen a heatmap under casino of gasses and delegate to reduce SQL queries to just think of it as soon as the user would question that is on the right we sat there money can buy dilate map-making facts they on internal cash so when you press the value of from a year is the data that is sometimes the wrong with our success factors that are a prospectus models like this on the of the DOA always start of the Kennedy Air Systems for suffers solar system which is that we only have we been only have 1 type but have more than 1 value and we'll say it we balance amongst them an unknown Opelika which is a few which is a new next we try a fault they will routing that will use it's quite stupid them in the sense that they will be just the round robin and randomized at the request of of Tyler's but the brain would have that is that when you it was but I yeah that I would paint all the top of the edges and vise but that there would not be so that would not serve the edges and dies because from running would produce some 1 0 1 so we use then that they collect 4 times demand of ties wanted to paint a without they far nothing so we went exploring hacking around will find out a of interesting with the this which is this and passing this is said that there it's you with this this is assigned like has at its ending in which several servants but with Indian accent opened resting a which is our dialog environment but on top of new method you can use to hook into requests OK you can decide what can use to to calculate the hash and so we went on and after some extremely we came up with this this simple piece of gold we just this a mechanism of operations given how quot 3 words to make sure that all the ties that have the same area that containing dissimilar benefits and so on and so by the same file server so it's tantamount optimal routing for for this with environments will 7 times and then again in the last thing with done to to play with a smaller therefore station on the performance of the selling was uh bitonality delicately it's the default transport format if all busses wall which is a citizen 8 by floats for its coordinates to a off their off its position for examining when they have a polynomial with any 100 with points for the corner vertices it will it will transfer all its those with users with approximately a position of max a loosely never have that position them and you need to have some of them at position and you're currently associations if you do then did you could do it the quiz manifold about but there we said to explore how could the changes and we ended up with something what we call tiny when a binary which is uh a specification that we open source and we want to build pop up on they and we work with some other people to do this and it's it's equivalent to that when a binary it but it's nothing going by position the mixture fits the the model as the the is that you they want to is Benatallah usually of the don't want sort of positioning for display up to 5 6 . 2 5 6 times usually with having way in which in which pixel suffixes that there is a point is and when I can talk about this in a in a later session but study 1 of my neighbors he helped us a risk we style of performance of this and a huge performance improvements because in new network in this case of this map was on 1 of our main bottlenecks and we just more we define when a binary and a thing you can get some these laugh when we moved reduced to our world traffic to temper some of of what it was on the traffic between the that the data was surrounded by the server a so yeah this is how lot of it in fact and it up like this so in constant improvements over a stack and that's all I think I'm a
very so if you have any questions thank thank you for your request so my comments right no
questions but no status of the