Spatial is not special: architecting for high performance geo

Video thumbnail (Frame 0) Video thumbnail (Frame 999) Video thumbnail (Frame 5686) Video thumbnail (Frame 6839) Video thumbnail (Frame 7635) Video thumbnail (Frame 8896) Video thumbnail (Frame 9733) Video thumbnail (Frame 11513) Video thumbnail (Frame 12306) Video thumbnail (Frame 14192) Video thumbnail (Frame 15385) Video thumbnail (Frame 17542) Video thumbnail (Frame 18090) Video thumbnail (Frame 18723) Video thumbnail (Frame 19756) Video thumbnail (Frame 21636) Video thumbnail (Frame 22683) Video thumbnail (Frame 24358) Video thumbnail (Frame 25024) Video thumbnail (Frame 26023) Video thumbnail (Frame 29771) Video thumbnail (Frame 30677) Video thumbnail (Frame 32483) Video thumbnail (Frame 34017) Video thumbnail (Frame 35417) Video thumbnail (Frame 36622) Video thumbnail (Frame 37486) Video thumbnail (Frame 39262) Video thumbnail (Frame 39839) Video thumbnail (Frame 41724)
Video in TIB AV-Portal: Spatial is not special: architecting for high performance geo

Formal Metadata

Spatial is not special: architecting for high performance geo
Title of Series
Part Number
Number of Parts
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Spatial is not special. Enlightenment comes from the realisation that a spatial index is just an index, a very cool one yes, but an index all the same. Gone are the days where the spatial database was the domain of a few large vendors and the GIS department. Spatial databases are everywhere now with almost every RDBMS and NoSQL DB supporting some kind of spatial index. In this presentation we delve into the realm of microservice architectures and containers and how applying the right tool for each job and how pushing geo-processing down the stack can cut response times from seconds to milliseconds and unlock the holy grail of IT architecture: unlimited scalability
Architecture Computer animation Lecture/Conference Software developer 3 (number) Species Smartphone Bit Electronic mailing list Variance Computer architecture
Purchasing State observer Presentation of a group Building Multiplication sign 3 (number) Mass Theory Power (physics) Architecture Goodness of fit Core dump Species Computing platform Social class Computer architecture Uniqueness quantification Projective plane Parallel port 3 (number) Bit CAN bus Message passing Computer animation Personal digital assistant
Computer animation Block (periodic table) Code Multiplication sign Surface Source code Table (information) Lattice (order) Physical system
Greatest element Building Mapping Building Gradient Denial-of-service attack Number Computer animation Network topology Universe (mathematics) Energy level Endliche Modelltheorie Hill differential equation
Web 2.0 Server (computing) Word Process (computing) Computer animation Multiplication sign Internet service provider Expert system Cartesian coordinate system Reverse engineering
Service (economics) Server (computing) Greatest element Gateway (telecommunications) Service (economics) Computer file Server (computing) Computer file Database Cartesian coordinate system Data management Architecture Computer animation Network topology Video game 5 (number)
Building System call Mapping Building Expert system Demoscene Web 2.0 Revision control Process (computing) Computer animation Personal digital assistant Web service Universe (mathematics) Moving average Right angle Form (programming) Computing platform Physical system Address space World Wide Web Consortium
Application service provider Mapping INTEGRAL Disintegration 3 (number) Library catalog Discrete element method Different (Kate Ryan album) Single-precision floating-point format Process (computing) Information security Computing platform Physical system Computer architecture Metropolitan area network Enterprise architecture Service (economics) Server (computing) Bit Computer animation Data center Computing platform Video game Website Information security Volume
Process (computing) Service (economics) Scaling (geometry) Database transaction
Computer animation Open source Multiplication sign Scalability Computer architecture
Scripting language Server (computing) Functional (mathematics) Scaling (geometry) Sequel Open source Multiplication sign Uniform resource locator Process (computing) Computer animation Term (mathematics) Energy level Video game Computing platform
Complex (psychology) Dependent and independent variables Rational number Multiplication sign Constructor (object-oriented programming) Coma Berenices Database Bit Disk read-and-write head Computer animation Vector space Query language Pressure
Point (geometry) Building Identifiability Open source Uniqueness quantification Moment (mathematics) Source code Electronic mailing list Mathematical analysis Denial-of-service attack Bit Code Dressing (medical) Number Category of being Uniform resource locator Film editing Computer animation Causality Query language Address space Address space
Mathematics Building Computer animation Green's function Building Multiplication sign Computer Mereology Task (computing) Geometry Address space
Complex (psychology) Service (economics) Functional (mathematics) Service (economics) Electric generator Link (knot theory) Inheritance (object-oriented programming) Software developer Electronic mailing list Data storage device Database Hand fan Architecture Message passing Causality Personal digital assistant Query language Videoconferencing Right angle Process (computing) Hydraulic jump Computer architecture Physical system Address space
Slide rule Service (economics) Overhead (computing) Multiplication sign 3 (number) Coma Berenices Line (geometry) CAN bus Message passing Cache (computing) Computer animation 4 (number) Query language Codierung <Programmierung> Computing platform Computer architecture Spacetime Window
Slide rule Electronic data interchange Archaeological field survey Virtual machine 3 (number) Perspective (visual) Power (physics) Pointer (computer programming) Process (computing) Meeting/Interview Lecture/Conference Personal digital assistant Endliche Modelltheorie Position operator
Point (geometry) Data mining Perfect group Meeting/Interview Lecture/Conference Internet service provider 3 (number) Point cloud Set (mathematics) Data conversion Shape (magazine) 19 (number)
12 (number) Vapor barrier Mapping Information Multiplication sign Archaeological field survey Lattice (order) Googol Meeting/Interview Interpreter (computing) Summierbarkeit Data conversion Quicksort Computing platform Metropolitan area network
Lecture/Conference Meeting/Interview Block (periodic table) Maxima and minima 3 (number) Cartesian coordinate system Perspective (visual)
Point (geometry) Metropolitan area network Addition Complex (psychology) Raw image format Functional (mathematics) Service (economics) Decision theory Data storage device Maxima and minima Control flow Bit Database 8 (number) Cartesian coordinate system Meeting/Interview Lecture/Conference Energy level Endliche Modelltheorie Partition (number theory) Computer architecture
Computer animation
OK so you're going to start our last session and so our
speakers smartphones and a developer but is also and and so is the father of Iris which is a geocoding inching but today is going to speak a little bit about my high-performance architecture and quality there's so let's welcome however speaker justice is
anything unique OK great it's good stuff and so on this is my 3rd foster G and I'm not I don't of go of the of the theory of much of its true or not and would like to test it briefly so I think that my theory is that people here fall into generally fall into 1 of 2 camps and why the marriage era people who falling in society and I think their IT people who have fallen into gear so a just a brief show of hands I know the audience who would class themselves as a G a person who has had to then I OK and the 2nd can is 90 person fallen into OK were quite this quite good splits kind of 50 50 so young I mean that in the latter camp and so I've worked for the last 15 years as a solution architect and my pretty much working exclusively in financial services in and quite lot that's time insurance and I fall into GOI by fitting to G about 10 years ago and I've worked on and off we GO projects overseas and times 1 going presented lazy it's nothing groundbreaking it's not rocket science it's just an example of something I've worked on it and I'm gonna take a very simple use cases and i'm gonna kind of work through uh how that's evolved over the years and I think where it's heading next and I think is quite a power is a few parallels that can be drawn to and GIS and solution of the solutions of architecture in general and so hopefully you'll gain something out of it and so you 1st of all the title is deliberately controversial and of course spatially special uh we wouldn't be here if it wasn't uh I have as as it has been suggested in the introduction just now in the way business over the last year uh that has a special technology the heart and eye on observation technology so yeah it's not special is special but it's not always special so from an architectural perspective and what I struggled with when I 1st started working with spatially and this has many new things to learn it's so meaning you have buzzwords and it's like likely have had this peak platform you need to do this and I I'm really looking to a kind of peel away the layers of the onion and really get down to the core of what it what it was the requirement what we actually trying to solve and I think by doing that so if you keep it simple and if you can keep it simple then your performance problems will what we take care of themselves and so that's my going to be my take-away message i'm gonna be coming back to that throughout the presentation so yeah having worked in insurance what insurers worry a lot that's what they pay today I'm myth essentially their professional gamblers and they gambling we've uh the gambling that houses all going to burn down and they can the
gambling that we're not gonna flights that they're of the building's not gonna fall in the grounds and that's what they're doing and they're they're basically trying to to make a profit for their shareholders based upon making good decisions and and and yeah and if they generate revenue than the that's great that's what therefore and obviously for us as consumers we want to know that if that 1 of these terrible things that happened to us and you know that we can get paid so afraid of the full pictures them in the 1st 2 a little bit more difficult to predict and the top 2 obviously there there's massive implications for geography and time and and it's no surprise that 1 of the 1 of the common biggest uh purchases of just technology is the insurance industry so for us
I come from the UK where we have a fantastic post system it's very very detailed it's down the and we don't like about a couple of blocks were block 50 houses 100 houses but and you that's but that's surface fine for a long time so juries have been working for
years we've post code table was and taking data from lots of data sources looking at the history of their claims to try and work out whether bad hot spots are and that was good and and and you know was doing OK but the problem is that
postcodes don't floods all burn down buildings burned down so uh as as as great as the postcodes have been actually will be insurers realized is that they they can gain competitive advantage by looking beyond the postcode and say what you think down to building level and our competitors is still doing things postcode level that's crazy it means we can give imagine we have a a street light that with the bottom of the streets of the river on the top of this tree is not you wanna give the guy at the bottom of the hill already bad price was saying forget about your mind sure you it's all but the customer the top is good customer would give them a good prices so actually if we can get down to building level then we will already gaining competitive advantage so what we can do this higher GIS this and smart people out there we got it employs a bunch of grades of university and get some
fantastic maps we destined by lots of lots of really expensive in number really great data and we're gonna build up models so we'll build up going back to the picture before we can build up a flat model we can build a house of subsidence model we can build a crime layer we can potentially look Austin and and where work at where we think buildings might burn down and a whole bunch of other cool stuff that we can do without so um yet that's that's what insurance companies would doing probably 15 years ago I guess when I when I entered via the arena
and because somebody happier happy business users and this and that's quite suddenly now we not working blind and we can start providing better prices for our customers but the problem was the
GI stains were becoming so busy are busy answering the business inquiries that didn't have any time left to do the job of the GIS so these guys again maxed out is then only quite small teams and and and and they're having answer these questions all the time so you what the solution is we can build a a web portal selects take uh the desktop experience in the hands of what the GIS thing we're doing on the desktop push onto a server a kind of a push onto a server the business users can help themselves fantastic problem solved and but at the time we didn't have any experience so kind of we knew when you have a desktop words but we didn't have any experience of of delivering these kind of applications the server and it's this GR ights really really complicated so you will call the experts in so doing we engage with idea very large provider of reverse GIS solutions to people I know of uh and said that we invited and the civil you need you
need a three-tier architecture and need this GIS application server and that's really important and you need a service lady but they to gateway and at the bottom you observe the still have databases and you and your files that you know if you do that lot that's fantastic and then the users will be able to look up and serve themselves they got
fantastic that the users are suddenly happy they consider themselves the skies a free again and they can start working enjoyable life scared
but another problem to the problem now is we have in the uh business users themselves but a customer on the phone the same right OK so we get we can give you a quotation but we still need to go and look up and see where you are on the map and OK right you look like you're in a bad flat area of but you're quite good answers to this process was still quite cumbersome quite long and the majority of the of the business that was being written was not going through this process it is just not possible and so but still the GOS platforms so let's let's take what we already had which was essentially a PaineWebber web version of the desktop the k and then let's integrate that with our from systems so if your customer home you getting your your Web quotation and you'll see in the universe the amount you not gonna go into usual building on a map of the behind the scenes what the what the that the city's solutions actually doing is he is doing a geographic lookup so it's GIS but without the maps the behind the scenes is going through interrogating all of those different layers to give you as the customer a price that reflects your house and not your neighbor's house the case so we we we
we did in our case we will get the experts in obviously now
that you know the required to become a bit more complicated so we need a much much more complicated architecture so now we need a B to the top we need our integration layer we need to be up to plot the solution into all the front systems and yeah well we need security would entail that we're gonna have all of this stuff sitting in the middle and an obsessive about and we have a desktop history can push up so we've got I all singing or dancing amazing GIS platform so uh yeah we're really happy OK that's great now so our customers a getting good prices and and and what's the thing good you know we we we writing less bad risks uh we getting less the claims and life's getting known much good looking pretty good for the insurance
company to but the problem is most people and if it is just again another show of hands if you get your insurance how many people would just go to either in the UK we have money supermarket we have aggregators like we have to compare the market then you when they can get their insurance basically just goes on to an aggregator website looks all the prices is the best 1 yeah I can exactly that probably around similar so most people now they don't go and sit there and look at every single different insurance companies they go on 1 of these great aggregators and it's so OK this is where I live bang bang bang bang bang take the cheapest but but the problem is that this fantastic GIS platform that we've invested in the data that we've invested in the team so we've invested in all of that stuff work in in our data centers great other day doesn't work for our aggregated business and that's probably 60 70 % of our business now so actually we've invested a huge amount of money and technology Institute but essentially giving a good price the 30 % of our business the other 70 % who knows which still taking a gamble when I was still working back on on postcards so what we gonna do what we can take that great enterprise GIS platform that we go in house and and justice Scalia out gets a little bit
tricky problem is what works well on a couple of service when your scaling out and you're looking at having to deliver uh subsecond performance you're looking to be able to process thousands of transactions per 2nd instead of instead of kind of 20 30 transactions a 2nd the thing doesn't scale anymore and then if it does
scale someone's going to be unhappy probably the CFI is not going to be very pleased so really what we do I mean where
do we go from here I K we don't have a scalable solution that we've got all of this very expensive proprietary tectum technology sitting in a way to we got what we think goes to the open source community that goes some you some fantastic offerings that have come out for some of the people in this room absolutely and that definitely
the way the way to go that she this time we need to maybe take a step back and think about what was the business requirement what we we trying to solve do we need to have all of these layers in our in our architecture had just
the will if we think about the requirement we just want to know the risk of bad stuff happening at a location uh do we need a matter not really no not for this was that we don't have the luxury of time for we just needed as somewhere that we can provide a an assessment of building level of uh of various different risk like a cake and we want to do that as quickly as possible for as many people as possible so
actually we could probably just be something like this so to instead of having this big platform actually all we probably need is a an API that speaks into and and said that that G adjacent adjacent if we if we not using a amount and that could just be a little Python now a simple little Python scripts job calling and directly and and you know what we don't need the idea that we don't need to have a yeah and j are justifications over we don't need all of these other layers if we took it down to the 1 function is to perform which is give me an assessment of these layers at this location that that's all we need and by breaking it down to something small like that actually now in terms of scaling that your life considerably easier so we don't have to even necessarily implemented in now that's a using don't that we could going use Python Wikinews Nigeria we use could carry on using sequel server if we wanted to we could go and look at an online open source offerings such as spatial Lytle or I will post yes and suddenly I you know of scale is reduced and now it's have to scale up something like this is a lot easier but we think we could
probably go a step further actually so even even this was still being able to perform even if you strip away the onion and get rid of all of the complexity in the midst here is still have to hit the database querying rationalize vector layers have to quite a few lookups and get a response back in and get that back crucially has to be subsecond and so you can actually call scratch their heads and full well do we need to be doing all of this spatial stuff every time we do a lookup sometimes yes but in a way that we that we can precalculate the answers if we knew the answers in advance we would have to calculate them for the customer
construction heads and had a bit of to think about it and then we
had a bit of a eureka moment we know where all the addresses are cause we by the data so it's a national dress status that we know the location of 37 million addresses in the UK we know exactly where they not and our customers approved probably only going to be looking get also from 1 of those 37 million locations and the other thing is what was actually at the moment were we doing we are mentioned at the beginning and that postcodes dying floods buildings floods with at the moment what we've gone from his post codes to points so at the moment if we have a yeah a flat cuts through the middle of the building and the entrance is over here but the layers over here and the majority of the flood is happening in the rest of the building actually we're giving an incorrect incorrect answer but because of the time-constraints in half into the G. did the GIS at a point of query we don't have the luxury of doing more complicated to uh complicated analysis if we go in and with this approach and we could potentially pre cash and do that answer list questions up front so what we do well
we've got a unique in the UK now we've just open source that uh you know together just source a unique property reference number say every single building in the UK every single address has a unique reference identifier is open source and it's available
and that's gonna take forever to analyze over that day so yeah how we gonna do that's gonna task and also every time the data changes it's going to take a long time but not not until
now say we can we can spilled the problem into chunks we could use a cluster we could if we wanted to we could use a cloud-based cluster and essentially we can answer that question what is the risk for all of those buildings and we can for again at dance hours rather than kind of days or weeks but the question comes up but we don't always ensure buildings so part of insurance is on a property developer just for this piece of land over there and I want reinsurance so that of C is not going to have a unique identifier and we're going to have to use the geo uh geographic approach that's the problem also that's the yelled solution for the 1 % so we end up with
something that potentially could like this so we've got out uh addressed wanted describe an address and for 99 per cent of the cases we can just do on identified but for the small percentage of the cases where we can't do that we go ahead and on a spatial database mobile-missile and I'm kind of what we've got here university without without once inserted to bring out buzzword is we we found associate that with a micro-services architecture we can go around doing it within their look on the line and see this fantastic thing and jump on the bandwagon and so we need to be micro-services which is kind of in where because we had just just by simplifying and solving 1 problem and and what we've got now is we something actually on the left here because as noted uh there's no spatial in here it's all we could use pretty much any technology that we want to still spatial over on the right this is powering the spatial where spatial queries on the left that we just really will just the key-value store so we can scale announced we can cluster and suddenly we're able to mates not subsecond that he would probably talking some 100 ms of 50 ms we've got something that's super super super fast so you actually another advantage of this is when their customers go online and and get a quotation and bang bang bang actually the guys who respond the quickest when only end up at the top so are not only that we we in upcoming further up the list and the other hidden benefit as well for the sudden as anyone anyone heard of uh experimented with 7 this architecture is quite a new thing yeah not wanted to label maybe on for another so that this opens the door for the 1st 4 7 it's architecture so suddenly now we have a simple function that does 1 thing the cause a key-value store evidence and data over my left there for people who on AWS customers I'm a big fan of AWS and we have a lambda service and we have a dynamo DB store and not being considered there and if it's not running it doesn't cost the city money it only costs us money when we use it so suddenly we have not only we eliminate Complexity we've got a faster service we can pretty much for next to nothing so the message really is that you free to use the right tools for the job and if you're interested in more these architectures is a couple of links that system that system videos and you chamber clips and we have the we're happy
developers and the guy
happy business users obviously this in silver bullets and it's not going to help you out with all of you really complex geospatial problems if you wanted to take your desktop architecture and there was published services very easy then great going user going into GIS platform but just when you go back Indian you're looking at this findings of questioning do I really need all of these lines do I need a full-function GIS platform do I need to be able to school every single standard or do I just need something that does 1 thing really well so to
summarize I would say that space is not always really special and you should optimize for your requirements the where platform and an approach the platform with that we with caution and we question the use of the platform catchy if you can't so if you are able to answer the question of violence done said then incur that said that that encode overhead at query time but above all just keep it simple and it was good and and that's my that's my message to the there you go to you and so I think we have time for a few questions yeah I'm sure they're going to be some yet we use the of the 1st problem 1 is in the nicest so you all we all just went back to the slide in question so you OK of plus
slides that's fine and essentially that the the data that we have a wish to receive were where come from From this year OK so the question was going back to the slides were show the picture of the UK in chunks and and how we distributed the processing and well he added that the way the data is actually delivered so you this that the data typically use comes from the ordnance survey in the UK and actually is delivered in an and in geographic chunks so that grid layouts that's the way the data is actually delivered so it's it's kind of already ready so it's already that positions already done for so we just take that great apply the same the same suggests it take to store the addresses from within that grid we with snapshot for example of model just for that bridge and and have that cued up and running on a machine and we can run as many of those in power as we was we've got the money for before 1 question on the not very well
necessary and what I'm I'm taking this and this and this is from my perspective now is a so essential and I'm running out this is assessed company now and say that the business already looking to see all business already looking to offload um uh any kind of use cases that are not secure and c
2 sets providers and this this 1 is this 1 is kind of perfect there's nothing there's nothing financial here there's nothing that would really warrior uh a big big insurance company and essentially this is just around them just around get different for providing geographic assessment so you and your i i didn't on unlucky died in has to have the conversation so yeah yeah this is this is more my taken as assessed providing you are free to use whatever technology I wanted but I'm seeing and I've certainly seen as a points with the possible but not before I was running mining company and who were totally embracing the clouds and embracing an and what services and so I think this is adopted is a feature is where the shape of things to come by what about yeah all you you
yes yes it is yeah you all yeah
and just just just a very sinister what if I found I consider your mom and the probably 1 of the 1st
conversations I had as an IT fashion having a conversation with a man with a year with the GIS person was our our interpretation of what was a map and you might similar thing so it sort of was published in a meeting for maybe an hour and and I'm talking to this guy and you know we we we we try we've been talking about this map and information that was the sum of the parts and in a familiar map is is what you see on google or the time or what I had identify had my own survey map and for him he was thinking of the at we talk about the base for the lady is although uh you know as I what what was the basement was like I just the whole of the 2nd barrier French found a really really hard and and I think I probably made decisions and and gave advice earlier on when I started off looking at this I just thinking as an outsider will you you must have to have a GIS platform you know that's absolutely how did you ask for the GIS platform so you yeah I think and I think it takes a while to actually break it down and simplified actually what what doing necessary is not necessary that complicated I don't know answer question yeah uh
when questions so from your perspective with your experience what would you think will be the biggest challenge if you want to transform like a monolithic application might seem to into micro-services so I mean you remove the complexity out of the the the small blocks but then you have like the orchestration yeah absolutely
using it's nice Stephanie no silver bullet any need to do you need to approach it with caution I haven't had to uh really
take something massive and break it up and my and the support the from from what I've written and whatever the that is that the hardest thing is is how you partition and what you do so this to the Führer micro-services architecture really there the corner guiding principle is that you should never shared a citation of a shared databases so you would have each and each function or each piece of the application would look at its own data store and so that's that's that's quite good way of segregating and but if you if you make a bad decision early on with special few just getting into this if you make a better decision you could end up adding huge size of additional complexities I think that would be would be a bit of a challenge and there's a bunch of approaches if you do have an old monolithic application that you want to to to kind of break is quite good said to think about the services the service layer on top of it 1st get back level we are happy with it even if it's just calling the modeling and then after that and that's the point where you can stop petitioning off so have that makes as all the other so that considerable devices by Ken so any more
questions OK so let's think this speaker and also the other speakers and I think this will be the end of the last session no less than the promises