Introduction to the geospatial goodies in Elasticsearch


Formal Metadata

Introduction to the geospatial goodies in Elasticsearch
Title of Series
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
FOSS4G, Open Source Geospatial Foundation (OSGeo)
Release Date
Production Year
Production Place
Portland, Oregon, United States of America

Content Metadata

Subject Area
In this session we'll introduce how you can work with spatial data in Elasticsearch - The Open Source, distributed, RESTful Search Engine. We'll provide a general introduction on how to index spatial data into Elasticsearch, then cover off on using spatial query and filters, before finishing up showing you how you can visualise and interact with spatial data stored in Elasticsearch using Kibana.
Computer animation
Point (geometry) Area Email Open source Wrapper (data mining) Duplex (telecommunications) Multiplication sign Connectivity (graph theory) View (database) Projective plane Bit Number Data mining Query language Search engine (computing) Personal digital assistant Average Elasticity (physics) Right angle Data structure Family
Implementation Service (economics) Divisor Connectivity (graph theory) Scientific modelling Solid geometry Event horizon Scalability Goodness of fit Term (mathematics) Natural number Database Elasticity (physics) Units of measurement Physical system Software engineering Mapping Closed set Structural load Content (media) Bit Representational state transfer Instance (computer science) Line (geometry) Demoscene Table (information) Subject indexing Computer animation Internet service provider Order (biology) Vertex (graph theory) Quicksort Data type
Point (geometry) Subject indexing Mathematics Mapping Polygon Elasticity (physics) Data type Field (computer science)
Subject indexing Pi Linker (computing) Quicksort Client (computing) Instance (computer science) Row (database) Library (computing) Connected space
Complex (psychology) Musical ensemble Open source Connectivity (graph theory) Real number Water vapor Shape (magazine) Complete metric space Goodness of fit Bit rate Strategy game Causality Videoconferencing Cuboid Elasticity (physics) Software testing Extension (kinesiology) Subtraction Units of measurement God Social class Default (computer science) Process (computing) Military base Software developer Surface Moment (mathematics) Representational state transfer Instance (computer science) Set (mathematics) Arithmetic mean Message passing Word Computer animation Query language Personal digital assistant Quicksort HTTP cookie Data type Row (database)
Subject indexing Computer animation Personal digital assistant Polygon Shape (magazine) Parameter (computer programming) Social class
Goodness of fit Military base Data type Metropolitan area network Resultant
Process (computing) Information Personal digital assistant Website Bit Quicksort Distance
Computer animation Query language Computer configuration Mathematical analysis Fitness function Quicksort Distance Local Group Resultant God
Metre Point (geometry) Subject indexing Uniform resource locator Summation Computer animation Query language Buckling Personal digital assistant Counting Data type
Point (geometry) Subject indexing Query language Reflection (mathematics) Physical law Counting
Numbering scheme Computer animation Mapping Angle Lattice (order)
Default (computer science) Subject indexing Sign (mathematics) Computer animation Open source Query language Projective plane Sampling (statistics) Configuration space Distance
Boss Corporation View (database) Scientific modelling Multiplication sign Projective plane Moment (mathematics) Electronic program guide Control flow Event horizon Chaining Computer animation Speicherbereinigung Right angle Quicksort Physical system Row (database)
Category of being Subject indexing Dot product Video game Computer animation Representation (politics) Right angle Solid geometry Event horizon
Default (computer science) Computer animation Multiplication sign Moment (mathematics) Data management Address space
Filter <Stochastik> Service (economics) Multiplication sign Moment (mathematics) Projective plane Gradient Set (mathematics) Instance (computer science) 2 (number) Subject indexing Video game Integrated development environment Lecture/Conference Internetworking Synchronization Elasticity (physics) Subtraction Data type Address space E-learning
the stone tools and so on and on and on and on and so on and on and on and on and on and on and on and on and on and on and on and on and on and the start the better the stone tools and so on and on and so on and so on and on and on and and on and on and on and on and on and on and on and on the story the the back in the tools and so on and on and on and on and on and and so on and so on and on and on and on and on and on and on and on and on the story of the back of the stone tools and so on and so on and so on and on and on and and on and on and on and on and on and on and on and on and on the story of the back to the stone and so on and on and on and on and on and on and on and on and on and on and on and on and on and on and on and on and on and on and on and on and on the story of the back in the stone tools and so on and on and on and on and on and so on and on and on and on and on and on and on and on and on and on and on and on and the story by the
the the the but not the size of review about as anymore the whole point of the war children both on and off and on and so in bars the of the of the knowledge of you know on the basis of of the area of all of us so the these sociology and of the book English-born English and by the end of the little brother Villa of ideas from which is good governance the the and don't all the from a to and the this the the sure that to get started before a failed coup OK so in for doing my name's Simon of and this is drawn with from a company called duplex based in male than in Australia in last year or so we've we've done a couple of projects which uh of we doubled in Elastic Search so if all be a nice idea to come to share our experiences and just go through a little bit of an introduction to the geospatial goodies inelastic search and can ask the rooms has anyone use toxic searches and ElasticSearch user can occur alright so what are
we gonna talk about and I guess relative us exert what is and what it probably isn't and we're going to talk about how you get spatial data into Elastic Search spend little bit time and 3 some queries and then briefly the component and which is like like it's like a dashboard for elastic search and also something that we've built on top of cabana as well I guess we're just starting out so it's it's pretty early days for us with it and was being good friends of all um worries ElasticSearch it's not source distributed RESTful search engine or as a friend of mine likes to call it a fancy wrapper around the same time yeah it's so it's pretty cool it's really good interrogating data so in both structured and unstructured and 1 of the features that we like about it is you can analyze data really really officially with it so from a structure point of view we can ask a question like give me all the e-mails sent by Simon on instructions I give me all males were assigned mention Edward Snowden's and then what is the average number of e-mails on rights relating to Edward Snowden and not all right many males will interact with so just in case anyone's anyone's things listening I would like to come back to america at some point and so it's it's really good at at at getting data and and finding interrogating and uh and it is a very very quickly which is which is killed and
elastic search is built to be always available and to scale with units so Ravil scalability comes from horizontal scaling and the ability to add more nodes to the cluster and to spread the load and the liability between the elastic is distributed by nature it knows how many uh sorry it knows how to manage multiple most provide skill in high visibility and orders running it uh is a running instance of Elastic Search while cluster consists of 1 or more nodes with the same cluster and so if you spend and you know would uh and defined the the simplest in uh as all the nodes Aurora you the master node will essentially reach out and grab that order enjoyment to the cluster as knowledge about a 2 or removed from the close to the cluster reorganize itself to spread the data in the and the good thing is that you really don't need to care too much about this stuff unless you sort of in the devil states side of things and so I don't really care about it but it's quite important if if you if you've got a cluster that that needs to be of all time and for me I was 1 of the solids it's it's a really well documented RESTful API are not very good software engineer and up to a factor of never used the term to describe the so software engineer and but but so but when I do Hackett stuff I I really I rely on good Doc cool and it's a really clean interface and on the docks agreed so to me that's not that's a big 1 suffix search and promote previous experience with things like solar it's just very little sullenly since been a little bit less accessible than what ElasticSearch is and then and the cost has special goodies which is kind of important for for for you and for this audience the what some the but before we go on a look at those goodies because it's that's fair you know our system or it was last extraction west it's not a GIS and that it shouldn't really be considered 1 that's not something make maps with so 1 my talk about and we don't really use as a persistent Stolzl so so what I mean by that is we kind of just assume that we can spin at all FIL data and drop it off as and when we we need to so in in the work that we've done we kind of assisted up to posters and and and then move the into a new Elastics search just to serve a particular purpose so we don't kind of consider it will not rely on subsistence it's facing not the answer to all of your problems and we kind of just use it as a component in amongst other things because it to be good at some things so so yeah it's kind of good to keep that in mind rather that we talked to a few people mostly when smallest excitons kind of sometimes is a belief that you just kind of user it's gonna answer all the problems that that's not it's just it's just 1 piece of 1 place of the further the puzzle so what well
when you wanna there with a lot of event data so I we're using it to to to deal with this event data that were receiving from from vehicles and when you want a better search experience and so important in a bottom line for me is because searching for stuff not often sucks so in what is being the game for about 10 years and and and still sort of say really really good implementation of searching across maps and and and there was a 6 might be at and we also think it's it's it's got a good done discography when you service on demand and obviously when you wanna do with unstructured data it ElasticSearch August would consider itself in schemas but behind the scenes it does it does have a formal schema in a mapping all topic of a 2nd the so um discovered talk talk through about get how gets that spatial data in and if you if you if you're in the room and you've got your RDBMS baggage review and indexes you can think of those as this as databases documents of stored in indexes and model as judicial and of the church of something called types so indexes content types of documents think of these tables and it has it has a single mapping and which is the definition of a schema definition of a but it's actually automatically inferred by Elastic Search short so if you want a slum in a document elastic actually have to worry about a mapping most social 2 will do the heavy lifting and create the mapping for you or automatically inferred from the from the data that you pushing however notes that's slightly different for geospatial data and content types data type fulfilled in documents so important types also a dual point and and and you should we'll talk about so basically you
at you create an index in this case on on creating an indexical accidents and on and on I insert the
mappings of ID fold ElasticSearch doesn't automatically create the correct field types for geospatial data therefore we have to explicitly create a mapping which includes the correct a special point so if you put at adjacent but you'd use and document which define polygon and this is the Elastic Search by day 4 will create a mapping with something like a double title for the for the coordinates for you to you can basically pull out the mapping and then changes in put it back in and and that means that you are successful then treat the gist geospatial type correctly so none special features would work if if you don't do this the kind of not Faulkner around mapping thing is is quite quite important to get to work and then
find a just world record so is just basically adjacent Doc that were pushing in an uh in the suburbs into an index so that's it's a subtype uh it's it's got ideas able and it's just basically edges and or of just remove the quote on sort pretty pretty much at some however you probably
probably you know you 21 wanna look for some 2 links inclined to and help you get the data in so bad reviews a couple things 1st on the talk was pies and which is a Python clients that validated resemble basically create a connection to ElasticSearch instance and then you launch data in use use in the library all you can choose whatever whatever you pour poison is there and for those of you recall fusion that's then you'll be thankful that the cold fusion point from an and so I think I'll
handle drawn who's gonna talk about rivers which is another way of a common data inelastic and and have careful with so we do was is the rate at ElasticSearch water people get detained Search and Bayesian it does document the suspect it goes so suppose that it goes on to run as a process so I Ileši? set specifies API developers call and their depending on that it's source JDBC 5 something else data about data there is dual of the dual component that consumes looks at the data and puts it into ElasticSearch ElasticSearch then you get the planet to running in the cluster and the best different strategies that you can of annual leave and so that's a static us this test all static value 1 off the initial pool and invest pioneers like every to means polluted and elastic set you see it so that's classical it goes the problem the the way the the the the the bodies did to get enough so don't you see if yeah when such that the problem with the videos of his source of each runs in the same cluster if if if the the the the will plugin that uses crappy it could take down you cluster your instances and the last thing doesn't come up so this happened lots of it because it does in the same JVM saved as about it would take now the other thing is it affects the elastic that performs as he should be released 20 huge sort of data it's going to London stadium and it's going to yeah greater of the so the new thing that they have introduces corner of long statue uh advance of the process you light in different data box altogether the default back in full of statues ElasticSearch again we a lot set specifies you can create equals from 80 any type of data and that put it into a real extension of sessions together pumping tho I just accept him and use ElasticSearch at all of the the the with that using a moment is I did this era of the founder of of developed for which runs all the process and the what on you know about you as a pop so this is the geospatial if the data that Siemens late now we have in indexing last excited and they're correctly tight in the just mentioned but ElasticSearch assignment mentioned has has really need that a CPA and the documentation for it has will daily you get combined could if you don't but did the complex queries aggregation surface at the 1 of is that all of this is document it's it's really hard to go into the aid of predicted that of aquatic like but of what good because it's so much in there you need to go into the local and the use of it from the rest the types of queries so there and they're like need to use it as you on the types of cookies as your he didn't go just about geospatial stuff so all the geospatial and quality capabilities all implemented as tests and the 1st 100 talk about users is that you shape could have and and the goddamn on you
could be in a simpler but just a quick word about what I'm you see here it's a called of light in cause since this allows me to do good paper queries to the elastic says Vest APA pretty easy to do so you don't have to worry about this like you don't have to compose goodies ladies you'll be using ElasticSearch plant of this class so you don't have to get scared of this this is just easier for me to for more this this is the this this case what I'm passing of good last cues like and here's my shape do the documents that you have been in last such that there are a that intersects this the it just like that and it through if you see of them makerspace the higher ladies clear does from 2 thousand 401 documents October certainly segments not such musicals pretty their thoughts and then by the 40 . against any records and testicle complete and units of bases have just because God but whatever going up and and now you have been to the cells and we're going to so you get back to the documents that the last such that intersect the shape that there I use specify the next
class P index shape also you can put your polygons up being index then you need us except then you don't have to quality them by specifying to the voluntary you can just before it blew the shape at ITU or the name and then asked asbestos this size so OK I already at U. of bond accord something of of region Cornell but do you know the arguments in no it's pretty easy to it in real
so in this case and the very so I know that the shape that I have passed I 2 9 3 good executive that's going to give me documents the dead and this is the 1st the are there
another type of bases you distance for day you could guess discrediting pretty good news that's investment documentation and you have so we don't good glasses it give me
I going to pass origin bright and asking documents at that obviously the 50 to the readers of the man less successfully give you those results the that would were the Knicks fun because again is this and I forget
about it's the same sort of thing I distance and specifying sort this is that this distance could becomes more useful you can ask the question and get the uh this uh sorted by how far they are from your place in evolved from the origin that you specify
I in this case have lost will be I have passed in the body and there have lost to order all sort might possessed by you distance and a bit of information that's so here a sordid unpiety so the closest 1 is 1st and I'm just going to move down the site to close to it so it could see that yeah so just on job this is but they go up all the last
exodus go like combining a few dozen qualities and now creating nested queries about this is that you can ask all sorts of interesting questions and sure distance aggregation again is like interesting question we have last will give that
so the elastic fit has been food does the ASCII ElasticSearch or yourself up the like God documents we don't be shy a 8 tool but that's that's leaderless has caused it to buckets to build buckets of to you but that led up behind it meet at 100 kilometers and the 2nd 1 is an option is minus 100 to 200 meters and that the war did the group by uh of the DFT it doesn't mean the size is 0 the don't want the results the border analysis of that's that's that's that this lasted 30 so useful in like an antiques you my best work that iterations that's to be read or give these other buckets news articles that's proceed that's pretty fast and I think
of OutputCollector gaze of OK so different concepts here I like classical the question you would ask from elastic is what you find the documents of the stable like Majesty's sums up here but the had been used the book and but the index of queries for us it index next squatters make up 50 meters from a given location at a query 1st and then be ask a question if document let true for their quality and that this is useful for like if you have a duty to like in
the case of the real data then you have if you reduce your point you cannot it is sitting there this dual friends your offences you'll be index could because I'm just asking uh so we have already set up by a of what you need to know and so it creates the index scored buckling and this uh a step you have you have to be there are like you need to we need to put his 1st so we have already done it and this suburbs use our percolated index and that the question asking is that something about this point we and and it's telling me OK despite what matters is the sum of sold if he had different types of what is set up you would give yes dispatch suburb and there'd be this is the count of accidents in that some of stuff like that in that's so that's
that's all the the query capabilities of the criminal law but not solved by the kinds of available data to look at ElasticSearch he added ElasticSearch cluster i and for later stages it's going to be consistent with the and just go go up popular but last a popular blind in the users would hate
that in it that gives you a lot of of your ElasticSearch cluster gives you the hair and you get out like if you had multiple Nordsaat the like named in funny names and listed on here I usually go for both had to check that really indexing you can T-bond refreshing and you see the count going up it was interesting plastic like the from from the point just oculesics and you can start asking queries select if if yes it up your documents or can be like exceed to start have assets was detained those documents become available facilities so that's it is pretty particularly so I just like in isomap indexing I just look at the reflections of see if the document are coming in another thing I go to check his later but is
that they have now and like if I want to inspect a document and the of
another 1 if awards chicken schema then I it gives me the scheme
and what's the mappings that it has created so this is I of angle to look at this and if I want
to do a as such was a few gaudy lattice expressed APA or the other quarters that's document you can put it here and you can try clitics so I used I'm going to talk about about Giovanni's but
again now open source project that's like the dashboard motion for elastic says began in index today they last exerts by the photons 9 213 out John saccule to if local host to by now it's picks up the necessary without any configuration this is the default that this is the default dentist this is look blue thing began create better and save it up I was I'm just using the sample data
uh dashboard here now you can stop asking queries and their that's also so panels that you can configure and people and get this this distance of continuing to of international good positive sign of still and just to finish off are in the we've talked
about a lot of talk about just pretty uh home of the components such as that of other of the project that we we basically use ElasticSearch on 2 we build out a sort of an that system
uh to to essentially montage more monitored garbage collection and so basically this devices on the truck which a measuring everything about what the trip is doing and thinking bark in millions of event records every day and what we do is we we essentially soaking all about data and we aggregated open something that that gobbles is Australians Coleman was called garbage guys Spain we call obj guys in garbage guides there so many garbage guy basically looks at the starch holds and then he can he can with a garbage guys that are that are that are lower down the chain so this is all about basically we're being uh employees to to to do things they don't normally do and also like invest a huge amount of money in in in the fleet like a garbage truck million dollars and what these guys who were driving garbage trucks and do is like a write them into the ground right is totally destroyed so essentially we've got all of this event is coming in and it's been aggregate Innotech search and so this is like the moment of the more time is in in in Australia the moment it's very early morning so there's not a lot of activity going on and off we look at it but also you have a driving RC 2 4 to 1 week of 15 counts of excessive breaking so also a garbage garbage boss will be on a chart with garbage to about is excessive breaking later on to today as a week was really we monitor all most of all the engine and also all the collections and now note this sort of the the the the model here is that the that's too bad garbage boss but is also the council that essentially all mandated to pick up the the trash and so this is a different view of the
world and so again this is this is life and life activity so it so so in the Council and and just as it of but you all just go back to just because also called right solid blue dots representative of of Mr. Bean's right so basically the beam was impressed was presented at the property uh so again we're what we're harvest data the common
of work out of the and so let's to take a butcher's this stuff so here you got missed being event and essentially what happens is the person that lives at fault pom-pom here present where the bodies and the 2 tends to ring that tends to bring the council's I'm sorry indeed include me being uh so basically what the what the Council do right is this a although we would you live or you live in a unified public resin right we were right your property that to 2 will fall in the afternoon and there's a camera shot of your house and you've been wasn't there so again this is now with the now within the customer when I and the customer instead of the thing got everyone's a winner by everyone's a and and and and what we've also done is we've also an index all the addressed it and so on and on its very
quick to to basically do things like just
property and and find out so um that's pretty much my time out of global management going questions about cost excitatory stuff from so at the moment we're doing very little special stuff with this and we just started out doing like classical aggregation to build all the defaults and also doing the address
the address uh look up and so our hope is to do more bring more geospatial filters we've talked about in in this environment or the or the projects to mystical that a higher and move your indexes you keeping next 4 by month and had you is the last 30 days and things like that you have to take the consideration a time of the dawn down the don't create it does seem exploding and that of the so but they have docket like different types of we have different types of documents inside and some of them you must have something to when you're using our s very and you sound successfully gradients and services that are secured also in In this instance it obviously engine x at the moment also interfacing in space so that like in would be static or something to get enough of that depends inspects the qualities ElasticSearch sits behind in the next nothing's the internet classes such as not directly to the Internet is any a major reason you don't use us to search for persist data and then during pro tips were keeping ask search in sync with 0 the stock but it's not the documents so the phone like 8 the you and what we are seeking up the ob but elastic sets doesn't love dates of because it's this in the sea it has to drop on a clear not days in document for deletion but seek the use of both the introduces Ileši? has decode bank to leave so you can lock a document pertaining to leave select if he knew so it's fine 5 seconds just want and the last the document that's so any other questions that are as much what not that sanity and was applied name new life and you're could rely on you for telling me at work


  462 ms - page object


AV-Portal 3.9.1 (0da88e96ae8dbbf323d1005dc12c7aa41dfc5a31)