A COG In The Machine - Using Cloud Optimised GeoTiffs to Query 24 Billion Pixels In Real-Time

Video in TIB AV-Portal: A COG In The Machine - Using Cloud Optimised GeoTiffs to Query 24 Billion Pixels In Real-Time

Formal Metadata

Title
A COG In The Machine - Using Cloud Optimised GeoTiffs to Query 24 Billion Pixels In Real-Time
Title of Series
Author
Contributors
License
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2019
Language
English

Content Metadata

Subject Area
Abstract
How do you find one pixel in a billion? Cloud Optimized GeoTiffs are a new standard for raster data that support file-level access via the internet. Combined with serverless cloud technologies, raster data can now be queried by client-facing applications without the need for a spatial database or specialist server software. In this talk I present how we used COGs and serverless to build a fast and scalable application to query large raster data using point and polygon geometries. As well as providing an overview of the solution architecture, I’ll explore the challenges we face with large raster data and why we chose to develop the solution using these open source standards and technologies.
Keywords General

Related Material

Video is cited by the following resource
Loading...
Category of being Slide rule Personal digital assistant Real number Multiplication sign Software developer Virtual machine Speech synthesis Cloud computing Pixel System call
Personal digital assistant Cartesian coordinate system
Category of being Uniform resource locator Building Process (computing) Mapping Information Network topology Website Denial-of-service attack Cartesian coordinate system Address space
Service (economics) Process (computing) State of matter Military operation Uniform resource name Cloud computing Set (mathematics) Integrated development environment Abelian category Windows Registry Address space
Metre Point (geometry) Scale (map) Presentation of a group Information Image resolution Range (statistics) Cloud computing Set (mathematics) Real-time operating system Client (computing) Event horizon Windows Registry Uniform resource locator Voting Website Integrated development environment Endliche Modelltheorie Extension (kinesiology) Physical system Spacetime Geometry Address space
Uniform resource locator Query language Cloud computing Website Database Database Cartesian coordinate system Computer architecture Address space
Covering space Metre Pixel Information Image resolution Set (mathematics) Database Bit Vector potential Type theory Different (Kate Ryan album) Quicksort Endliche Modelltheorie Pixel Computer architecture
Information Divisor Multiplication sign Denial-of-service attack Real-time operating system Database Subset Uniform resource locator Crash (computing) Term (mathematics) Personal digital assistant Website Extension (kinesiology)
Mathematics Euclidean vector Independent set (graph theory) Raster graphics Function (mathematics) Multiplication sign 40 (number) Projective plane MIDI Core dump Extension (kinesiology)
Area Laptop Presentation of a group Context awareness Multiplication sign 40 (number) Cloud computing Special unitary group Mathematical optimization
Area Pixel Information 1 (number) Website Cuboid Cloud computing Database Mathematical optimization Product (business) Point cloud
Axiom of choice Reading (process) Pixel Service (economics) Computer file Cloud computing Set (mathematics) Computer 2 (number) Product (business) Revision control Geometry Web service Internetworking Core dump Cuboid Pixel Mathematical optimization Address space Area Constraint (mathematics) Closed set Cloud computing Bit Machine code Line (geometry) Open set Connected space Cognition Uniform resource locator Process (computing) Raster graphics Function (mathematics) Object (grammar) Spacetime Geometry Library (computing) Address space
Gateway (telecommunications) Reading (process) Curve Dependent and independent variables Functional (mathematics) Scaling (geometry) Multiplication sign Data storage device Cloud computing Real-time operating system Bit Database Mereology Dressing (medical) Front and back ends Connected space Number Arithmetic mean Film editing Process (computing) Function (mathematics) Object (grammar) Computer architecture Address space
Area Service (economics) Computer file Military base Run time (program lifecycle phase) Multiplication sign Electronic mailing list Cloud computing Similarity (geometry) Medical imaging Raster graphics Different (Kate Ryan album) Blog Revision control Damping Software testing Series (mathematics) Compilation album Lambda calculus Dean number Address space
Multiplication sign Service (economics) Intel Pixel Standard deviation Metric system Service (economics) Mapping Dependent and independent variables Multiplication sign Moment (mathematics) Cloud computing Cartesian coordinate system 2 (number) Sample (statistics) Raster graphics Personal digital assistant Software testing Physical system Descriptive statistics Address space
Area Flock (web browser) Building Polygon Denial-of-service attack Website Energy level Denial-of-service attack Office suite Machine code Address space Vector potential
Scale (map) Surface Pixel Satellite Denial-of-service attack System call Compilation album Area Product (business)
Real-time operating system
Standard deviation Functional (mathematics) Service (economics) Open source State of matter Multiplication sign Decision theory Moment (mathematics) Electronic mailing list Cloud computing Knot Cloud computing Coma Berenices Open set Machine code Web service Different (Kate Ryan album) Blog Internet service provider Ring (mathematics) Routing Spacetime Form (programming) Address space
Computer chess State of matter Multiplication sign Function (mathematics) Client (computing) Open set Perspective (visual) Front and back ends Medical imaging Mechanism design Different (Kate Ryan album) Software framework Office suite Endliche Modelltheorie Data conversion Moment (mathematics) Binary code Data storage device Physicalism Cloud computing Bit Connected space Wave Process (computing) Internet service provider output Right angle Quicksort Geometry Spacetime Point (geometry) Functional (mathematics) Service (economics) Open source Virtual machine Random matrix Web browser Drop (liquid) 2 (number) Number Product (business) Force Latent heat Term (mathematics) Energy level Software testing Computer architecture Scaling (geometry) Surface Plastikkarte Basis <Mathematik> Machine code Integrated development environment Personal digital assistant
so it's now time to ask the was holder is going to leave his speech about the call when the machine.
at that. i. what. but this is also an. the idea of it. yet so thank you get my as thomas and i what i just found and only the technical development of us solution and haste talk about cloud optimized year to this and how we use them which is a bit different some of the other use cases that we've seen so great to see so many talks about cogs at this for forty. so really briefly one slide just cloud provides joke owning and risk intelligence for the businesses around the world and primarily in the financial sector and with often sure is that we can tell in show us how risky a property may be for underwriting purposes.
i saw quite a key markets and we also work as commercial markets as the deliveries on the sticks so today i'm going to focus on the use case of risk profiling so this is our maps application.
and in this scenario an insurer has has dropped a pen on the map and has turned some information on that we have from the data about flooding and floods verity we can see that this this location which doesn't have an address it's a greenfield site and so we would know where is geographically but we haven't been addressed and attached to ok. in that location has a really high and flood risks we can seize gets got really really high school so in short you very unlikely to want to write that location now of course we've got an awesome data from our data provided us about about floating about subsidence.
when storm about trees nearby and then intelligence about the building itself obviously that's a large amount of data and are back and so we need to be able to quickly that information really very effectively in for a very very quickly so that we can tell i'm sure enormous no time at all. what the risk is like to be about property and so all of the process is that we do i feel i let so without some applications we have an a.p.i. applications built on but also some of our customers connect to and the documentation is available docs are dressed out dot com if you're interested in the things that we do.
when the kinds of data sets that bring together.
and so work and it wasn't a partisan have varying data requirements but obviously most u.s. data comes in one of two flavors will know better and faster and so we have an internal process of taking all of those states acts together and i'm pulling them into our back and into our service and then allowing people to queries absurd.
this for some information.
but those data sets are getting bigger because increasingly the resolution of the data sets is improving now how this will serve as models in the uk down to five metres so that's the resolution of the foot model talk more about locked in a second and when some of our clients connected to our system they may be. competing in real time against other vendors so in the uk we have aggregate aggregator websites if you go to get your house insurance or your car insurance. you may then go to compare the market for example you see a range of prices that are coming back to real time so when that happens i'm unsure asked us to tell a tell him about the risk of not point in space we need to be able to respond to them very very quickly typically so one second and not keeps me really busy because i have to be able to think about how. duty i asked was really big data that's it really good resolutions but after that really really really quickly and that's that's my of my age of what i do. should also give a kind of really big banks because this presentations an extension of some work i first presented at your mother in london which is a really great event a thing is now one in berlin as well so if you are any the vote to locations it's really nice crowd of people to come and hang out with them and talk about geo things so.
so in the beginning there was the database and of course it was post just because we all have purchased in post is awesome. and we know it big post just back and to pull a status together and do spatial assets that we need.
so have not location and architecture that looks like this so i was a database sites where all the data is have an application that we've written that combines queries all the data and then we have an a.p.i. an application from yorkshire will most people in the room that done some.
spatial development will become a familiar with this this kind of architecture so this is a bit of an overview of one of the types of data sets that we've got from one of our data devices that the flight data so that dividing us to this model and which is a set of five meter resolution six twenty four billion pixels just of. the land cover once you've must out the other thirty billion pixels that are just the ocean to oversee not bothered about and. the sort of the and you think i'm sure remote sensing background you might think about rematch of deaths is actually a fifty two different layers of potential information for every pixel not data set and it's quite quite a lot of information.
as i said before we have quite strict plays with lots of our customers where we have to base the divide them the information but only a subset of it not the entire dataset we have to be able to clear and say for this small possible and this is what we can tell you in terms of the flood risk.
and and so then that's great but then what if we want to other countries like north america and like home around like are a our post just database bill is going to get really big and to be able to like scale up and ingesting this stuff with posters roster being able to carry in real time should mention everything's cashed anyway. a we have to have use cases where the car crash because it is a greenfield site so we don't know in advance what every locations going to be so what can we do the same time it's been a put this in because there is also talk about what was happening the future posters roster and while i was going to go and it's been a factor down to separate extension.
so it's really good to get some updates this morning and see that project is in fact up but still kind of popcorn popper she says and is continuing but for us it was kind of a time of well do we need to look at other solutions how we can empower going to make this thing this thing work i'm and luckily i was the last forty.
most topical alex lisa who is here and i think is just giving a presentation who was talking about your packages and mention this thing called coxon as the first time i've heard of a cloud optimise year to live.
some guys like wrote made some notes my notebook diligently thought and filed away and few months a to face this problem as like is this the sun us the specification aware that her about the area for forty.
so enter the cloud optimise years if i'm really pleased that i hope lots of you've been to some of the other products mice years of talks because it was a brilliant ones that get really good details about how a car going to file and under the hood works so that i don't need to only to cover the information and directly myself but really quickly can say no cloud optimise year to fish.
just a way of internally tiling and the tips information and is accessible over the websites a cloud first and data products that means you can say for this bounding box just give me those pics of bodies and first is amazing because that starts to transform the to file into a career of all searchable database if you can say. now i want this area just want a small handful of pixels don't need the other twenty eight billion just give me the values for those.
so not not too much code hope you can have just barely legible. and we can actually use their really roster a library that was created by my box to directly interact with locked out of to my status so if instead you through this is the only lines of code i'm going to show but so are going to import and astray and imposing and then this this is the magic this is where realize that this thing. for us is going to be like gold because i can put my cloud optimise year to file in what's called and asked the book it which is of objects or from amazon web services and i can make a connection to that file in that book across the internet so i don't need my computer to be in the same location as my data still. and then the second bit of magic as i can say i've got some core of us and coronets and geographical space give me the pics of ali's the fall within that space and i can do that straight are eighty s. the book it really great now i've got out maybe ten pixels coming back which is a very small amount of data and something quite quickly and only to start to worry about. let's not worry about the size of my budget of one of its off constraints of using postmistress to us which which is the kind of the previous version of this was our retail process was getting longer and longer as our data sets are getting bigger and more data sets that larger areas more areas of the world and so that was one of the problems that we face that in this version because the day. to provide up provides us the data set as a to file such you quite an easy and a simple process to convert up existing geo to into a car optimized to to. honestly i can read those readers pixels out and i can send him back to my the i can continue the process billion so now is kind of got some pieces on the border front of us i'm stuck to put these things together we've got our data in a bucket we can access that we know that we can pull off to see a.p.i. what we're going to do in the middle and well we are just cloud.
so everything we do is is in the clouds cognitive and asset base this year will be hundred percent service and to support our capability to the natural choice for us was to use of another amazon web services product called us address lumber and and under is a piece of technology allows you to a close in code.
the in britain that is executed in response to function and can execute as many of them in peril as you want to his company is scalable and when not when not function is finished running it shuts down to new pay for the time that runs typically you may run that cofre maybe one to two seconds. fantastic so we can drop in u.s. london the middle of got our data stored the back end that data storage costs of an object stall outcomes s three mean that even not fall which as many many gigabytes doesn't really cost us anything to store idea of a slumber function and does it really cost us anything to run and we use in another.
cut from it from a person can take our gateway provides our a.p.i. not doesn't really cost anything either for this this surgically part of the architecture so all of a sudden we've gone from what was becoming quite a big challenge money a process of floating data into posters and then making sure that our coaches databases were able to scale with. and in real time without uses low latency but high number of connections into something which scales effortlessly and this is very very cheap which is great and there is a little bit of learning has been a learning curve to get the restorative package to work in a dress lumber now thankfully boxes.
around again. and that they did a blog post a billion baht person talked about how to take us to a package and some of the underlying disagreed on of the but some and put bases a doctor image of a spy and doesn't have to compile listings from scratch and get rid of a lot of the things that you don't need a lot of documentation files get rid of all about releasing nothing down and the there's a. but with a london layer that means anyone can can get this now and upload as a base in a series of compiled by some files so and some get have anyone can stop using it with example i show to stop playing kind of playing around. and the service time as brilliant so we did with his test this is one hundred similar tennis requests for different areas of one.
don't really big master of those requests are all coming into the service same time in the meantime was seventy four million seconds and the star description so deviations ariz twenty milliseconds so we're really happy with that because it means that we can be we can begin that service and as much as one weekend it's just going to do all day just going to keep giving you got those pixels first as long as you want. and so our overall status continue to be the same way the doing some some performance testing at the moment so excited to see what happens when we start to scale this and more than one hundred requests when we continually this movies like a soap test and waiting thousands of requests over a couple of days. the bia hopefully writing such presenting about not in the future so i kind of wanted to finish and have ever use case to put this in really kind of give an example shows a map suggest this is a great news case of our application and the stuff that we do with cogs so this is a caravan park and my colleague mark as new.
they were used to go on holiday when he was a little boy the seaside in england and so if we do joe code on the address for that car park the postal address wear them with a male those is i'm is this the red pain economy sale but is that the front door the main office the building that's bricks and mortar. but the risks to the insurer is obviously all of the caravans and across the entire site so the flood risk for the main building for the office where the post to live it is one so very low out i think about writing as and shore but their insurer is actually ensuring all those carbon so doing the site ensuring all site so what can we do well. we can allow are in short to come to accept occasion and draw a custom three hundred polygon around a site and say i'm underwriting this these caravans of this segment of the foresight and if we turn on the third less as we actually see that that the area as a whole has got very very high level of potential putting.
so probably isn't such a good thing to to want to underwrite we've got that close call this comeback on the left this come up to thirty is a compilation it's been done by polling all of the pixels that are in in not in the product last year fifty and adding that so score together to go and normalizing it and to view about you.
and so that's what happens in real time i'm wearing and then we can also visualize not by we actually convert outs my status into a stylized reputation using some textiles and so that actually could turn on as a player you can see ok that's a river running through the outside so happily it was.
quicker than i expected to run a blog post that dies into some of the details about this have blocked addressed now dot com so you can you can check out and that's it i thank the coming three states was into this session last day and i had a really good conference and it was really great to me all of you and hope to meet a few more soft and in fact so much.
and three months are there any questions. you. who knew previously said that you land all your moved to london. so is the only reason why you did this because it now looks like your date's tool and amazon for it. i. so we were already using amazon web services to deliver deliver our services so we already using it so it made sense for us to use that technology to to provide the service designed to question. yeah. don't you want to get another question. not really but your if your i was curious if you really want to use amazon and if you want to move to go to plan for example i think now it's quite hard for you so i'm curious why you take the e.u. to this decision it's and yet because at the time we started working on not this service but other services. that use the same technology amazon was the only solution provider that provided the solution. you have under lock and with the cloud providers an interesting challenge one of things that we don't do it the moment it's using another it. piece of two in called terra form that allows us to and basically infrastructure is code so we can code our infrastructure so that helps because that supports a different route cloud services back and but we would have to do would be a really really are easily major effective to move some of the stuff over of their some really great initiatives in knots. space open function as a service and so i'm what i would hope is that over time we as a native west london but the mice such a sport open standard on when we can take to take advantage of the year. thanks. the other questions. the sharia. it is. well i really wish i'd like written a list of things that light your to work on and ready for next year. at the end of a fuss body conference so the question was what was my wish list for the open source of his facial community and i think.
clout to my studio tips and the tooling that's been lost is emerging around a really good example of thinking about gea spatial data in us in a service environment in a cloud first environment and i'd like to see that continue because and architecture that we have is not a you could not replicate it with a more traditional seven baseball. it'll on we would not be able to provide that service that we do with the latency is that we have with the number of users that we have with without an enormously large number of physical machines and even then i think it would cause problems so continuing to talk and prosperity is one such twenty winners of a good conversations going. on in the community both online and in person about how we continue to move your special check into the cloud and also how we continue to embrace our you know open source nus on open source the thoughts and take that with this and start to challenge some of the things about why can't i run my functions in different car providers. and technology open. and isn't question church. and. smart. i know. care for no questions such as just that and just going on to the kind of continue from but thomas saying around the love and looking up at a kind of had that in the past about been looking i think was two things i think one i'm i think if you try to be clad agnostic ormeau to cloud i think you missed the point. and you miss most of the benefits you get from a particular cloud if you convert the resort have that level of a level of the attraction and then the other things while i don't think you look yourself and the lander sent you an infected simplify your coat down to some pitino some javascript and actually the bits that make a lambo okubo functions. so as you have a so minimal that actually the ri right to move for months now that would be would be quite small in and then you can always use things like yet we didn't really get on with the service framework but you can use extractions said his final tell terrible was as to mention to kind of take that way so yet don't think though be don't be scared about going on in on one cloud. and nothing to move to another one would not be a massive massive he shouldn't think. but it. really hard to seeing such a bright lights shining down on. a wave wouldn't and on one was inside the bus this. it. it. it. you are. yeah differently from the to you please read the show us the question was also talks about big bear to think out of my status in the browser and and told us that this year to ask been presented this week to i see an opportunity that to to work with a code in the back and think i was a question. question. the u.s.o. first i would say we're not you were not rendering talks cox on the client and using them as for us to a back and data stored terrible thing that year to chastise are just really good and it runs in knows it could go on the back and some quite interested to performance test against the stuff that we don't apply for the biggest problem we have a moment is compatibility some of the lower level binaries stuff. what's going on in python and get not updated so that years of chess for example is a p.o. javascript imitation of the country's not instead that might be that might be really good so and in terms of tying i don't know of no would not have not experimented with looking at stuff on a on the front and so i'm be interesting to see whether out where all this kind of. based goes. the. any more questions we do have time. right. he says. up. the. the. i. the question was for n.b.c. the classifications with the help and on the browser or service side and i think it depends on the use case i think that some brilliant examples of doing stuff on the client but for us that we everything's a.p.i. back so we need to be able to push out that data that possible. keisha know whatever is the a.p.i. so that's why we're interested in what can we do in the back end. because we wouldn't want to replicate something where we have one process and africa and clients thought not available through the a.p. i am so you know we're a paper i let it as many customers using an a.p.i. as we do have been using the front and tooling of so i really think it depends on the space and what you want to achieve without data think the examples i showed her. we're quite different to some examples lots of most examples of seemed cogs are of our g b imagery and so that services completely different use case what we've got enough a synthetic products that this just using geo to offices storage mechanism but it's not just in its doesn't wouldn't have to doesn't have to be an image for example but that's because it's a continuous surf. this data products to the best way currently to store thanks. right to own. it's more questions you want to. but you've got to fit on a business that sets his handlers and so my question is are you so if you have caught like the coverage of all uk is really want image really two hundred gigabytes the cock it could have done. i didn't even say that when it first came from the data provider like ok and yet not there is so that the works with the performance are shown so it's really impressive from a cool again has only just try out of its kind of pasta works because every penny we know where to go into this mess big a.t.l. cases of chopping up. put an end. yet it seems to be enough back and environment we do do some stuff around cashing the connection so making sure they've always got on that date is always warm as its as it's called to the access to the dating the book and that the code to the and queries that data is that those things are always ready to go that helps quite a lot. yeah it's seems um yeah i'm really impressed that the whether the magic is and i put half people in the audience to thank the contribute to that specific o'clock specification the magic this and there is a pretty phenomenal from our perspective. but we do have time for one will form one more question. i. and i need to exercise some of it. the. along the same mime as the question that was ordered coming out there was already asked do you first see like you've got a two hundred white image and they do for see issues with large images or some sort of performance drop off for something like that already. think it would be linear in terms of performance now i would want to quote up so late said i don't know back and say that it was probably not going to base its not going to scale to say like north america. going to have to start to think about what we do that but i have been just us or so for example no the data provider is doing it in north america is doing on a state basis to provide you with or over what you use to call the region. because i think the that they would run into problems creating the input data and their output data that we're taking and they have problems crane and that's what model to to to fit within that you continue surface so then i think then this of interesting things i wanted to do we take all of those and tyler and so i don't know something that we're going to work on this year. the. it. i. a lot. oh. oh. it. now so the question was unarmed as and have a stop the time of one or two or even longer a number of seconds and how do we deal with that so we have always been s.l.a. we deal with that by making sure that our monitoring service our intelligence services or is continuously polling are. service to make sure that the the sandbox worker which is the piece of infrastructure that amazon has that checks out that lumber code and runs it to make sure that some walks what worker is always around and the kind of listening those can persist for over the course of the day. the same work would persist so so say i wasn't and so if that things always around and your the start time decreases and so you have this concept in the technology is called stop and a warm stop and so few the biggest i show in the customer's connect to us and make it clear that the in. numbers which award already to the side of time is just reduced the two months thanks to his.
Loading...
Feedback

Timings

  349 ms - page object

Version

AV-Portal 3.20.2 (36f6df173ce4850b467c9cb7af359cf1cdaed247)
hidden