Bestand wählen
Merken

Server-Side Marker Clustering For Rapid Display of Large Datasets

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
well I figure out people trickling will just go ahead and get started in see with that goes on they surveyed showing up my Johnson and the talk about a server-side worker clustering for rapid display of large datasets on some of the fun problems we ran into in some potential solutions and where we'd like to take forward error bars not here a scene is not to people in just the 1 of the key got called way of meeting so I get stuck with doing the presentation of my own
so what what I the the project that started this whole thing I was a website that was designed word to be designed and to list out historic places of end of specifically in uh community in northern California and the project needed to happen at a number of different things of mainly a map with points on the client wanted the website to be able to be on responsive Hasenyager moderation and as well as administrative side that you can call like everything away and we see that uh initially with public data from from the community around and we light weight and there is not a large amount of cost that could be involved with over and over with that of
maintaining the site and keeping it running and and responsive just then asked for but which would have been that just for fun on we
jump into the demo and at any point if you have questions shout out I'm kind of just we should always around the and
look for any kind of feedback so this is
that the historic mapper history
mapper Isaac tell we've a 4 hour demo insight testing we've moved it often to Boston not into a northern California and with large datasets on we initially
started with about 600 to 700
points for the the initial site and those were not really well by just ship and then do you Jason of and AJAX call over to the client side and it would do all the marker clustering using leaflet marker cluster as a familiar with probably hopefully maybe and it works pretty good slaw browsers got a little slow so we ended up moving to a different model where we would send out on the data as it was of as they zoomed around and changed James extents and this work pretty well the mark ring out a little flickery but it was OK until we got past about maybe a thousand points on the amount of data that we're shipping with G Jason just in time over the wire plus of browsers being not the super computers that they are on all computers from didn't quite run as fast at a low level of model and time so
alone theory Boston with the marker clusters and as you zoom around its grabbing more more data as we get into a certain way far and here the the there we go
on as we zoom in further over on the left side we get along FIL tiles start showing
up that allow the user to click CAT and the thumbnail of of what there is you can click on it get
more information courses you can now view details and part of the user moderation side of this was that
of a moderator can edit places and the public can suggest places suggest changes
so you can also make comments that here I made a very nice yeah building comment
and then it places is much like you would expect most at once the look
like the I gives the status of the the
overall that the project a root of the place you
can add delete the keywords and of the idea of to worse of about 2
were is just a colleague at a distance of a hybrid and keyword
durable to selected and then it
will just select those
of particular places and search using the basically a full-text search 4 of the the back end is Mongo DB and
we found that the full text search it was also kind there is there were a lot of
1st that happened with this project and it was ah companies 1st dive into a full open source of project and it was our 1st dive into no 1st dive into Mongo alone and so that presented a lot of interesting challenges but it was more so just an awesome way to grow and learn and we had a great time with on we use AWS uh as the hosting platform for the the box and S 3 for storage and it was my 1st dive into using their SES forcing out e-mails which meant no SMTP which is pretty awesome following we built on a virus scanner because you know bias is a bad island so I came out to where we claim maybe to scan the buyers were scanned the file as a as it was uploaded I said earlier we did use leaflet and the cluster marker plugin and so it started getting a little sluggish after about a thousand features or so and so that led us to start Japan don't come back that led us to start looking at other ideas of how to use the tools that are in the community as well as maybe you know I can think outside the box a little bit likely somebody can always thought outside the box for us and uh the the beauty that we suddenly realized of using no JS is its script and you can use the browser of libraries as well so so we search around and found that ended up finding out that there's the server-side leaflet the own package and it is a package that essentially when every fired up it mimics for leaflet on that it mimics the browser DOM and so it is able they have the full functionality of leaflet plus the the features of the plug-ins that can come along with it so we use the server-side leaflet to push the geo Jason data into leaflet and then we basically look and loop through all the zoom levels all the markers and save those out among go so then Mongo has those and when the the the user changes in levels the changes extents just the clusters are now pushed out so instead of pushing out thousands of features at the beginning of the map years pushing out 2 3 4 5 features whenever they move and then more as they zoom in further so it down we we created the the cache of clusters it does stored mongo I using the dollar sign the Mungo spatial function dollar geo within yeah on we're able to pass in and extent of the view and then I get the data out really quick is a what were cashing it up on the on the server so it's a nighttime job that runs on and in rebel to you work were trying to is a lot go in a minute but we're trying to help to go were trying to get to work more of a real time and clustering of because the user experiences in Austin where you have a new point of it somewhere way out in you know that's not gonna clustering doesn't show up until you zoom into the last using levels not ideal but I do what we worked with we're pretty plot please what came out of it as I said earlier I I got to learn the Mongol and node which was exciting and really fun the fun job and and and the other thing that really made stuff easy was using the get public hooks to whenever I pushed out of our code and it showed up magically on the server and restarted the node process and did some other things made sure that the in the in the in packages that were added some were updated as well and brought in so kind of an automated deployment fashion for the and so overall we started looking at how light weight this is we've got it running on a micro instance up in the for opinion is on and the flexibility of mongo and the flexibility of a node in general with how easy it is to change schema modify you know all your data handling and were looking to take the same idea in the same framework and to other projects such as may be modified around for asset management field data collection so and utility management a lot of what we do we work with a lot of public entities and there always looking for a cool light weight weighted to handle a lot of GIS features we would like to get the the RAM reduced of it essentially what happens is because you loading and in in our Boston demo we've got 130 thousand points other were downloaded from the ministers that that the historic society over there and it takes a while it's of good 10 minute process but because you run into the nite it's cool and it does take in about 4 gigs of RAM which does not running on our Micro instance so what were looking to do and is if we forget you can find a way to get this too much lower will aspire up at nite fire above you know a bigger instance for 20 minutes but a boot run the cluster show the back down the beauty of Amazon and pay for 20 minutes of server time and we've looked also at dropping the clustering altogether the cluster caching altogether and moving into GeoServer and it would probably take it take us up Greer server bit I a little bit more RAM but we've looked into this going straight to GeoServer so there is there are some Mongo around to GeoServer plug-ins connectors we've seen 1 I played around with a little bit I got it to work OK so so that's kind of also led us to porting over to PostGIS 2 other storage mechanism but as I say here a relational DB they're somewhat more familiar to CIs admins in the in the in the public sector they like the fact that they can look at a table and you know they can look at relationships and they don't near the the whole idea of document based and no sequel based is it out there pretty fast on but moving to post GIS would have the added benefit of really easy access into GeoServer on so we've got a couple things that were looking forward to moving up with and and changing around and the really interested were were not uh that were really good tool uses of and I were really good tool actors and finding them and try to make more and so getting into the world of kind of caching on the fly would probably have to be something that like reduce server the but were also really interested in trying to find out better recall cooler ways to do mapreduce is inside of Mongol itself and see where we can go with that so I think very
time I appreciated of and I'm pretty quick and it would be pretty fast pretty painless of any comments or questions
the other ideas did I miss something that was out there is a 1 of the in the so the in
the web storing them in Mongo of and storing them just as the geo Jason object so it's just a and so it's just that it's a of properties in attributes of words at attributes and it's geometry and when the program gets the the that adjacent point back out I have a type field in the property and it tells whether it's just a marker that's living out by itself it's not actually a cluster and so render it as just the blue and teardrop some of them Europe if it's a it becomes with the type of cluster of miles of account found and so it's just come across as a little bit of just enough information to draw it come along with the convex hull that comes out so you can have the same kind of there's the same kind of ideas as the market cluster of and loading it you know the whole dataset was just cashing in and streaming out those clusters but place on you hear the
sound of 1 of the churches that's right yeah that's a good good idea and like as some store
the the convex hull it's going to be pretty easy to do that the called very by a set of
this what you have to use the we did l and on a number of the which are the number browsers my by my colleague Eric has what I call the boat inca computer and it is really old all the way up to year which I don't know higher super fast ones and so on but I chided because that's where the project started I I started with the idea of Kerala my grandma's computer and and we were seeing it load times whenever you got about a thousand features we're 2 to 3 seconds on the server side and the the server side you're looking about 10 minutes or so at 100 K about 4 gigs of RAM and for the whole process now the cluster part of it on actually goes on fairly quick Italy maybe takes about 2 3 minutes but I've got some other things that I'm doing I erately and now looking back on it would love to change it
but I didn't store my data in Mongo my raw data in G adjacent format so I've got a converter and there is a process where I have to convert from my storage format to do you Jason so you the 1st and most I just found some of that last week on in in trying to get ready and chartered think about you know how I can move forward with the project I I just started find some of those of more you know stuff they 2 leaflets node cluster clustering the yes on sorry I probably should be
passed around microphone because the people a string can hear you but he had a the 1st was
asking the that the timing of the the clustering on the server which is about 10 minutes or so given everything that goes into it and then the 2nd question was if I had considered using other there's plenty services K type that yeah there's a couple of other bound algorithms to do the clustering basically completely outside of the flea market cluster In the back
and the I hadn't that's an interesting that there would be an interesting no way to go about it on might 90 at under no had considered that's an interesting idea because then once you get zoomed-in you could just turn up the heat map and show the ROC points and then again you're only string and a couple local oneself to another that's another cool idea of on that would go really well if we to poured over toward you servers well to make that really easy and so on and so on focus I did I got a chance to play with a little bit on again my
with my data not being energy adjacent format not really a standard for I was just you know Adam putting a schema together all my encountered some issues on definitely you know over you know not insurmountable but on but I kind of looked at that as well what is
and what kind and you know how it has on yes so on the on a graph
I understood the question right it was is there any clustering happening on this on the client side itself not all the time had a of a lot of of the that's a good point I on since the data is fairly static of but you're right it does so and every time the they extend changes another of AJAX request is going back to the server so caching that would be on the client side would be smart at all questions that and that is what the I think that's a that's a very good very good point cool cool any other comments or questions again type a few more research and yes sir well I and II GeoServer has 0 mean and he doesn't know GeoServer prior better at the point what's the what's the of function there is basically a does style that you can apply to a layer and it will at what that that's right . stagger thank you thank Yuri much iodide I knew was there zones of my time on but yet . stacker of looks looks to be a very there are some weighted tried to to do that and to move forward and I think that if I got good enough with my styling it from making even look like the the leaflet clustering as well such I used to be in the hands of the part of the of the of the of the of the of the of the of the of know that that that was of something that 1 of those things to there they are really cool that the market clusters is an
amazing and at the and died that they animations are pretty slick on and I do think they help the user understand what's happening as well like other point disappeared with you know what happened now on but I haven't I that's a interesting I may look into that there be pretty cool of it as it was being is that
that the product was originally released a lot of non GI answers I had to explain what the clusters were on and it was an inherently apparent to a to the general user but does it's words were really excited was telling that turned out so the will think you're much I I few
appreciate that all the comments and the ideas and it's also thank you
Demoszene <Programmierung>
Verbandstheorie
Datensichtgerät
Kombinatorische Gruppentheorie
Computeranimation
Fehlermeldung
Mapping <Computergraphik>
Subtraktion
Client
Web Site
Punkt
Gewicht <Mathematik>
Webforum
Endogene Variable
Zahlenbereich
Projektive Ebene
Wort <Informatik>
Computeranimation
Softwaretest
Rückkopplung
Demo <Programm>
Wechselsprung
Punkt
Konvexe Hülle
Elektronischer Datenaustausch
Web Site
Computeranimation
Web Site
Punkt
Browser
Güte der Anpassung
Systemaufruf
Web Site
Gerade
Computeranimation
Übergang
Informationsmodellierung
Client
Unterring
Supercomputer
Cluster <Datenanalyse>
Maßerweiterung
Konvexe Hülle
Einheit <Mathematik>
Parkettierung
Massestrom
Web Site
Cluster <Rechnernetz>
Physikalische Theorie
Computeranimation
Inklusion <Mathematik>
SCI <Informatik>
Stellenring
Sichtenkonzept
Konvexe Hülle
Mathematisierung
IRIS-T
Web Site
Tablet PC
Resonanz
Computeranimation
Gesetz <Physik>
Webforum
Authentifikation
Verschlingung
Zahlenbereich
Mereologie
Anwendungssoftware
Versuchsplanung
Flächeninhalt
Information
Sigma-Algebra
Computerunterstützte Übersetzung
Stellenring
E-Mail
Lie-Gruppe
Computeranimation
Wurm <Informatik>
Intel
Task
Bus <Informatik>
Anwendungssoftware
Flächeninhalt
Versuchsplanung
Wurzel <Mathematik>
Addition
Normalvektor
Inklusion <Mathematik>
SCSI
Elektronischer Programmführer
Gebäude <Mathematik>
Web Site
Kreisbogen
Inverser Limes
Zustandsdichte
Zahlenbereich
Ruhmasse
Hochvakuum
Projektive Ebene
Hill-Differentialgleichung
User Generated Content
Dualitätstheorie
Stellenring
Elektronischer Datenaustausch
Web Site
Benutzerfreundlichkeit
Computeranimation
Spezialrechner
Mailing-Liste
Hardware-in-the-loop
Hook <Programmierung>
SLAM-Verfahren
Zahlenbereich
Ruhmasse
Verbandstheorie
Hill-Differentialgleichung
Abstand
Gasdruck
Hybridrechner
Chi-Quadrat-Verteilung
Hash-Algorithmus
Front-End <Software>
Web Site
Benutzerbeteiligung
Bit
Demo <Programm>
Prozess <Physik>
Punkt
Browser
Datenmanagement
Fortsetzung <Mathematik>
Computeranimation
Übergang
Datenmanagement
Prozess <Informatik>
Softwarewerkzeug
Skript <Programm>
Cluster <Datenanalyse>
E-Mail
Caching
Lineares Funktional
Kraftfahrzeugmechatroniker
Sichtenkonzept
Güte der Anpassung
Speicher <Informatik>
Plot <Graphische Darstellung>
Zoom
Knotenmenge
Virensuchprogramm
Datenfeld
Framework <Informatik>
Server
Projektive Ebene
Tabelle <Informatik>
Instantiierung
Relationale Datenbank
Server
Gewicht <Mathematik>
Quader
Mathematisierung
Systemplattform
Räumliche Anordnung
Framework <Informatik>
Code
Loop
Programmbibliothek
Speicher <Informatik>
Cluster <Rechnernetz>
Maßerweiterung
Data Encryption Standard
Relationale Datenbank
Elektronische Publikation
Booten
Vererbungshierarchie
Open Source
Browser
Softwarewerkzeug
Systemverwaltung
Plug in
Elektronische Publikation
Ordnungsreduktion
Mapping <Computergraphik>
Echtzeitsystem
Caching
Cluster <Datenanalyse>
Bit
Punkt
Kategorie <Mathematik>
Konvexe Hülle
Räumliche Anordnung
Objekt <Kategorie>
Benutzerbeteiligung
Datenfeld
Datentyp
Wort <Informatik>
Cluster <Datenanalyse>
Information
Cluster <Rechnernetz>
Optimierung
Attributierte Grammatik
Konvexe Hülle
Vorlesung/Konferenz
Speicher <Informatik>
Umsetzung <Informatik>
Prozess <Physik>
Browser
Zwei
Zahlenbereich
Computer
Eins
Rohdaten
Last
Mereologie
Server
Dateiformat
Vorlesung/Konferenz
Projektive Ebene
Cluster <Datenanalyse>
Speicher <Informatik>
Dienst <Informatik>
Algorithmus
Datentyp
Server
Vorlesung/Konferenz
Cluster <Datenanalyse>
Zeichenkette
Mapping <Computergraphik>
Energiedichte
Bit
Punkt
Besprechung/Interview
Stellenring
Server
Dateiformat
ROC-Kurve
Fokalpunkt
Standardabweichung
Zeichenkette
Lineares Funktional
Client
Punkt
Graph
Rechter Winkel
Datentyp
Mathematisierung
Mereologie
Besprechung/Interview
Server
Cluster <Rechnernetz>
Zeitzone
Punkt
Besprechung/Interview
Vorlesung/Konferenz
Wort <Informatik>
Cluster <Rechnernetz>
Biprodukt
Besprechung/Interview

Metadaten

Formale Metadaten

Titel Server-Side Marker Clustering For Rapid Display of Large Datasets
Serientitel FOSS4G 2014 Portland
Autor Ingbar, Eric
Lizenz CC-Namensnennung 3.0 Deutschland:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
DOI 10.5446/31735
Herausgeber FOSS4G, Open Source Geospatial Foundation (OSGeo)
Erscheinungsjahr 2014
Sprache Englisch
Produzent FOSS4G
Open Source Geospatial Foundation (OSGeo)
Produktionsjahr 2014
Produktionsort Portland, Oregon, United States of America

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract What happens when you try to give map users a rich display with thousands of points in a web browser? Generally, one of two things occurs. One, the map looks like a carpet of marker symbols. Two, depending on device and bandwidth, a browser-side clustering algorithm can delay the display for an agonizingly long time. The problem is acute in dynamic map systems where automated processes or users generate content because the system architects cannot know in advance how many map entities will be created. Think of mapping tweets, potholes in a major city, or locations of coffee places in Portland Ð map displays have to show dense data effectively. Our presentation shows how a Leaflet plug-in is used on a node.js server to solve this problem. We will demonstrate how the Leaflet.markercluster plug-in is used to generate clusters. Once generated, the clusters are stored in MongoDB and retrieved using a MongoDB geospatial index.
Schlagwörter node.js
Leaflet
marker cluster
MongoDB
user experience

Ähnliche Filme

Loading...
Feedback