Bestand wählen

Dynamic mapping on the web: building a scalable service for thousands of companies

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Erkannte Entitäten
and I'm a homogeneous I work at systems of transitivity and well ahead there have been a lot of talks about already about the which is something
the easiest way to get the data in a mountain of public thinks they just uploading the data at being a CSP
and file an actual file and this in a public it's styling and get lots of analysis and insights for it and showing bisociations but I'm gonna talk about more about the insights on the the havoc is with the end to get this thing to work properly because we're from pretty fast unwithdrawn from being a small a system based on completely open-source a
stock and actually the whole country reveal or almost all parts parts of it are open source a unreasonable there lots of all so much from the open source community of boss's boss's mapping diesel and they're building blocks and we build on top of them on and also making them making an open source platform which we to serve millions of times every day to all parts on the world this is actually hidden not
that that from where people are that that looking all ties but the does have that we actually ran several certain of that those were actually serving or on the wall this is such a date that how bit effectively due to some in Spain and was even lot of dynamic data but of the magic thing about us because we're open source and the and everyone that anyone could replicate is pretty easy but the magic parts with half is
environment we just use through induced give us of our data studied and customize your SQL and the way you want to be so as to data I was so it for you and will be share everything keeps working will kicked a fork can right and in this regard a country these different to other solutions a thing because we're dynamic in the
sense so it to the economic as well as users to do anything you want with the data using the SQL API to operate the data and b and make it automatically in a brief rest and rendered again on on
in in on ties in this is not an example of an application made up by Fulcrum in which is a using a mobile app for gas inspectors to love where they the done inspections of gas infrastructure and the 7 habits of gas inspector on on the map that is refreshed and real density can see which inspections wouldn't yesterday today the further by using SQL in this case a single year yes to filter with with section of data during sitting and his his style and course with highly dynamic but a
so Islamic because a all information to can be updated in real-time at would be refreshed in real time but it also has to be aware of that uh important an important component of this dynamic part which is that most of the dynamic bottom that dynamic
maps are static in the late manager in this in this animated map at its there in the super bowl which is an a 1 of them off the hits number softer maps this was delayed the let the last remark I showing today there actually it is studied in so that the data doesn't change and and is estimated to millions of people but it then it never gets in that there behind it never gets you 1st it can be refreshed if they are of of the map this decides to put on another 12 at the more of tweets it would be rendered but they were actually kind of most of the maps and things were serving been sort of
static because if not they will and in the same thing 1 someone's so once again I would be just wasting to resources and then the self rendering of on rendering and change things and the amount that's where with us why winning a cross-layer update and
they are headed to recommit because they because they were their way we have it's a the any any moment anyone update is that they are
a we have 2 layers of passion 1 of those is there something that I the percussion ran on of vanish and then on this and catching on molten use vastly actually uh on hundreds of points of presence on the walls and that we have it goes to monitor a for example this
is our day was the following request for from 1 server and actually of this mass which is also the data were 7 we actually just
running of this and even though it that a lot of the of the data and do which is a cache misses come from not for internal gasses so we're saying that a lot
to to actually been able to scale maps to pass on to millions of viewers because actually they're written they're reads to write ratio and about 4 is very high even though we if you wanna write we will share a common right but it and right now we're focused on being able to read hundreds and hundreds of times actually even be able in my with air moldable contractions
become this 1 is is a bit of a scale triggering molybdenum bodies done in a Python inside SQL in is being loaded by a close as wanted to put it because it's looking like I love a contrasting with with that but the and the fact is we're actually bear flexible but we also have very control on what users can do but there were factually in in the sense
that any user can upload any geometries a half and then run any SQL query that gets or heart out of their heads heads to the operations with it and then when the rate using the bill render in linear which is on particle call which after get to that later a within its good and CSS you desire were given power to the the to the big things
so they can upload their own huge geometries of hundreds of megabytes of Paul against and there aren't any kind of displaced operations on top of them and they can render it a with any Haskell is yes as they want it and of course that this is from system side that looks up and that looks and it's a bit of a hollow actually because most of their damages that users upload that are not that optimize and they
got for example heat position for a new states they wanted out they don't want to see for example a uploading data with meters or sending the precision of a of a country for just a window where the door you for the country and getting I'm wasting space and their accounts with them resources and that as running at a special operations that pretty heavy because posteriors has some very good things but as some air renowned military optimize things and then rendering with correct rendering rendering by using the muscular most of most of the of the times it is not the real thing for the Duff but they will have no them anyway to tell it to the users so we have to live with it what with if with and but attractive been pointed that from trying to learn how a user so doing this stuff at keeping QoS metrics on every server database server every user in this this is
this this is our internal laterally have for a this is a not for the whole country platform
but for a subset of users we're keeping track of the of the Biot by insisting response time and I Requests on this clicker responses but it but the thing is that as we allow users to do anything they want we can
bound and be predictable and say all the queries will take less than 100 ms because that because sometimes it takes a lot more than that to just get this good results of the geometry ones around the so we're putting a s work actually close in the wall so all of these a flexibility because a the premise i in supporting because they
use a share a database server and apostasy seasons and busses hadn't has no built-in way to restrict their CPU usage or the queries are user wants to run apart from sentence mean apart from Senate by not so wasn't that long about for example 5 minutes there so it actually will connect to it with an finally and above posses which will allow not not only medium by the end by statement timeout by dosing is a Linux 60 should give the processes of the of the it's a good database connection a different priority on the database so we give them power but limit the CPU power they use they had learn how he a all the customers who and to get it out of a history of how we grow role up the that firm on this to explain that at 1st when were very
small and we were in the hands of users of of the Kennedy was just a bare-metal service which all that is all of the all of the parts and of the of the and things in 1 single machine the database their skill API which is an an API you can use it to justify throw always skill queries to the through this debate of a center of Belisarius with with that which is open source to which is ways of category which is was on top of Martinique and then our Matt management you i which is the more they in which is made and will be rows with a hefty lots of gas with guns at both and a what we if this just we use there you as we need to scale and the more we as a study showed in the users this is not the
real distribution but a sort of adding new service to the to the to the to the thing understood within service users between servers but that they had that solving problems the gets to be in it and that's a be that's a convenience when you have a single user which is a which is hampering the several lots and get into the the put into the whole potential that off the CPU of that server and then we have a lot of a lot of piling resources on all of the bare service they are so then and there was a bond would still have to scale
and we moved to answer me she to
uh and so on there will move what was previously inside a single machine which is distributed over machines so we can know share parts of the of the of computing power among the tighter the school the and the red side and actually we use them for different database servers done a thing that is unique to encounter the 2 into a new set is there that database server which it it which means it's user is shown in a class indeed replicated or not and then the rest of the thing the share the tightest and actually rendered by from all the machines this really go connects all the machines you will have some sort of priority prioritizing so enterprise and paying users get more resources and it did not get to the more queries and get that traffic break ties entire between the against a free users but it was from a that is the way they were were going for now on the path from foreign was you was unit CPU from the from the equation there is no longer a bottleneck
but when you start moving up things and it's in things the way they're not supposed to be a defined at a different there a different kind of problems which is the bottlenecks where that where that I find them to define them for example there there was a
client in which just several uh data from out Bloom on the relatively small database n which 1 of those times in this form for example this 1 is just empty that it actually is such a limit of a polygon and is when it's actually also think upon the data I think it there in the south west arrest rice volume and for some reason and we know notice that when we move from the from the from the dedicated instances and but have the same machine to putting out in this about things everything is article had those the same particular study to and not much more Mars slowly again victim figured out why so is that in the candidate set in motion of the the polygon that uh that those food ties surrender form is actually 25 megabytes of data which is a standard database and that was yes semantic airport they're cutting the polygon and intersection cutting off but assistance it doesn't seem to be very efficient so what money gasses is actually Donald the whole polygon and painted the proteins that 1 that 1 that I said do get in that part of of information from our big polygon but also it sounded like to a suicide to look at the parole that Bush AS users to communicate with money and resist all and as the Internet protocol to reduce which is the WKB
on father they all of the data that between the bosses 7 and the better service is sent and WKT which actually has an aid life float for each x coordinate a that it might be a the we're
basically transfer and all of the bonds of the polygon 8 but it beats per byte the aid which provided a London in that depend on the part for the coordinate over a single net will link so what happened we we just we wanted a monitoring tools we look at a traffic and we found
that the reason that without that was everything was going wrong with because we were actually collapsing the the networking the faces of machine guns and data it's would've it with the mask and the they just they turn out to and it turned out that for its file that rendering will transfer into the family of it's a failure which is all of the points of the polygon with the 8 by position with sexually abused enough 2 .
17 per cent of the power of minus 10 position and yet this so this was all phase 1 and find out and
at because of the fact that this data was looked a almost a always after they opt a almost everything on their which often is look at heightened levels are in really we high sitting in meters these things but it would be enough but it's a 10 to the minus 10 centimeters hundred like the funny too much
and with that in the there's some tweaks a with using a height a hacked up the Bession of both both CAS unmet need to actually use uh our formats we go WKB which is so it's a stupid form of it is just get the the the bites and get the precision to half and there's like 1 had this the new MIT not once in the near position that for high high levels is even less than a pixel and nobody will know is that and were we were experimented to its but a is of the solution there were actually a doing we want to the clean way and will work on real solution we decide to escape the bosses and then
at some said unsoaked some state so we get the some traffic to to kill with uh too many precision of a and B with quick gets a solid solution efforts on alone were thinkin is to just there is Posterous and and the and Tyler wish was thinking of of finding something to that backs
Idalia a and then passes it because most of the times that the users are not in the SQL or the source of the data itself at the changing styles and we don't need to get the the whole data from that and we will want to get away with not hitting that the data was never as much as we can and we're still working on that and you were 1 of the process where we investigate in yesteryears not neglectful ties at the output of the Bosphorus and then put gas in the mail so we can just AUC they use the same data for both rendering science uh using MAP need to and we just on the server analysis and in the same mapping aggressive doesn't to client by using guide rendering which is something we are also working on and there and this is 1 of just 1 of the lot of a bunch of problems were having the the scaling and their profound so adhering to see if a
with fellow we're we're looking for some people rowing we just that some money and they if you're in the sink there uh that would have a job states it with their positions were looking especially scenery relights had revealed the engineers and developers and all kinds of of the arch with glue in with good primes work with so that's all thank you and if you had
a questions as a time if
Hey what would thank if and it for 1 thing I do offer caching as I have all the data I Kapila geometries all the other data for each mapped into its own Cash table and then I send it to the child so these there's no some queries interested in you're like windshaft in How than is that faster than all them tell Stan Shih er have comparison old y you and with that because any right now using tell treatment have also yeah actually
uh well uh right there which might be more suitable than to answer the question that's a we just it was done nothing before dies starch so they're pretty compatible which shuffles develop in-house like to be in those open-source tool models fit the new requirements without but basically they're they're just a front to Martinique and the way it doesn't things behind so you in that regard as moralists a compatible that does that has the hots a lot of good things on which of costume which every just fill in our needs and wetter thinking on investigating how it looks because it it might be actually better than ours and about the casting the problem with a with this is that they do get a well the use dynamically the disco queries to the diff difficult to Koch the did the data output itself because we don't know how they that 1st there's a bunch of data and 2nd we don't know how often will the user release it because imagine he's just editing styles in the currently and we get to a 2 and still quite a fish things and we tested to have that that that would be quite inefficient so working and doing some market like catching on demand like Mark get passing the data that the user sector quest so when they're not inside the instead of upon embossed not they're not the whole data because if not will have to to cost maybe even bounding boxes are based on on the way that it does not the law 1 of your
last diagram showed post rescue something and VT and cash the caching layer for those rescue all their work and that 1 there was really words at the MBT was listed indicate and you'd see the sun intermediate
for a format which is actually being created by my parts which basically means in mapping vectors science which is just a way to get the time information of a tightening the set up a buffer me in a very efficient way and to get all the information related to a geometry so this was coming developed yet but basically wanted to generate the science because we we felt science there style adjusted their their geometries we can discuss the geometry is the raw geometries and then a painting and they're different in the Tyler are even in in in times the thank goodness who without the units
fewer this there's no question about the some of the part the words as the cacheable the MBT there's a for a new standard Build style cache of Cassia we're going Viennese time cache of us
will be the most the thank you
Gesetz <Physik>
Gerichteter Graph
Formale Semantik
Elektronischer Programmführer
App <Programm>
Befehl <Informatik>
Kategorie <Mathematik>
Gebäude <Mathematik>
Güte der Anpassung
Dichte <Physik>
Dienst <Informatik>
Einheit <Mathematik>
Rechter Winkel
Tabelle <Informatik>
Lesen <Datenverarbeitung>
Klasse <Mathematik>
Räumliche Anordnung
Open Source
Virtuelle Maschine
Weg <Topologie>
Endogene Variable
Elektronischer Fingerabdruck
Spezifisches Volumen
Ganze Funktion
Protokoll <Datenverarbeitungssystem>
Open Source
Anwendungsspezifischer Prozessor
Elektronische Publikation
Wort <Informatik>
Prozess <Physik>
Gemeinsamer Speicher
Familie <Mathematik>
Kartesische Koordinaten
Einheit <Mathematik>
Prozess <Informatik>
Funktion <Mathematik>
Nichtlinearer Operator
Zentrische Streckung
Kanal <Bildverarbeitung>
Twitter <Softwareplattform>
Gewicht <Mathematik>
Gebäude <Mathematik>
Puffer <Netzplantechnik>
Inverser Limes
Zusammenhängender Graph
Operations Research
Leistung <Physik>
Einfach zusammenhängender Raum
Diskretes System
Physikalisches System
Design by Contract
Mapping <Computergraphik>
Universal product code


Formale Metadaten

Titel Dynamic mapping on the web: building a scalable service for thousands of companies
Serientitel FOSS4G 2014 Portland
Autor Santana, Javier
Martinez, Alejandro
Lizenz CC-Namensnennung 3.0 Deutschland:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
DOI 10.5446/31752
Herausgeber FOSS4G, Open Source Geospatial Foundation (OSGeo)
Erscheinungsjahr 2014
Sprache Englisch
Produzent FOSS4G
Open Source Geospatial Foundation (OSGeo)
Produktionsjahr 2014
Produktionsort Portland, Oregon, United States of America

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract At CartoDB is an open source stack that includes PostgreSQL, PostGIS, Mapnik and Leaflet. The hosted version enables thousands of users to make new and interesting maps everyday. With some of those users including Al Jazeera America, Twitter, and even online gaming platforms, we aren't scaling for one popular webpage but for thousands of different ones each day. On top of that, maps aren't constrained to a single filter, single style, or to a predefined zoom, CartoDB allows users to access the full power of a dynamic database from the front end. In this talk, I'll present the architecture decisions we have implemented that make it possible to turn PostgreSQL and PostGIS into components of a powerful real-time data visualization tool. These decisions cut straight through the CartoDB software stack, from PostgreSQL and PostGIS through our caching and tile services, and up through to our CartoDB.js library. We'll talk about our on-demand tiling service, our caching strategy, and our implementation of the novel data format for Torque. Each of these areas has enabled our users to make user of entirely open source tools to create maps and services that scale, remain fast, and are beautiful.
Schlagwörter performance
data visualization

Ähnliche Filme