We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Simple is Better: An Intro to Event-Driven Serverless Architectures for Faster Disaster Response

00:00

Formal Metadata

Title
Simple is Better: An Intro to Event-Driven Serverless Architectures for Faster Disaster Response
Title of Series
Number of Parts
295
Author
Contributors
License
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
In a disaster or humanitarian relief context, time obviously matters. This talk will show how, without concern for the details of servers and storage, you can use small amounts of code to quickly build powerful solutions, using a prototype disaster response pipeline as an example.
Keywords
129
131
137
139
Thumbnail
28:17
Event horizonServer (computing)ArchitectureOpen setOpen setMathematicsDependent and independent variablesBitWeb serviceMultiplication signOpen sourceSoftwareRight angleLecture/ConferenceMeeting/InterviewComputer animation
Computer programProcess (computing)Set (mathematics)Information privacyMereologyWeb serviceOpen setPoint cloudComputer programmingFile formatSpacetimeData storage deviceMassComputer animation
Link (knot theory)Point cloudField (computer science)Theory of relativitySet (mathematics)Web pageWorkloadDependent and independent variablesChannel capacityOnline helpRight angleArea
Computer programDependent and independent variablesConnectivity (graph theory)Texture mappingBuildingInformationCartesian coordinate systemQuicksortOnline helpBitDependent and independent variablesSet (mathematics)Information privacyOpen setInformationPoint cloudGoodness of fitLevel (video gaming)Different (Kate Ryan album)Channel capacityGroup actionBuildingAreaField (computer science)Basis <Mathematik>Connected spaceCloud computingWater vaporTask (computing)Moment (mathematics)Computer animation
Dependent and independent variablesOpen setWindows RegistryQuicksortSlide ruleSet (mathematics)Open setWater vaporSatelliteLink (knot theory)Bounded variationVariety (linguistics)Group actionReal-time operating systemWindows RegistryDependent and independent variablesSoftwareHill differential equationSource codeXML
Directed setService (economics)Workstation <Musikinstrument>DistanceScheduling (computing)SatelliteInformationSoftwareDependent and independent variablesPoint cloudMotion captureProcess (computing)Distribution (mathematics)Right angleSoftware testingComputer animation
Library catalogCore dumpExtension (kinesiology)Focus (optics)Plane (geometry)SatellitePoint cloudVideoconferencingDigital signalEndliche ModelltheorieTemporal logicPoint (geometry)Process (computing)Subject indexingBuildingComputer animation
Element (mathematics)Entire functionLibrary catalogAmsterdam Ordnance DatumPrice indexElasticity (physics)Element (mathematics)Information privacyBuildingSummierbarkeitPoint cloudPhysical systemInterface (computing)Subject indexingSet (mathematics)Cartesian coordinate systemQuicksortRepresentation (politics)MetadataComputer animation
Interface (computing)DemosceneMaxima and minimaInformation managementWechselseitige InformationNewton's law of universal gravitationDenial-of-service attackServer (computing)Category of beingLevel (video gaming)Analytic continuationDependent and independent variablesInterface (computing)MappingContext awarenessWorkloadAreaRight angle
Web browserInterface (computing)Visualization (computer graphics)QuicksortContext awarenessMathematical analysisSimilarity (geometry)Functional (mathematics)Group actionProcess (computing)Server (computing)
Interface (computing)Server (computing)Lambda calculusServer (computing)Level (video gaming)BitFunctional (mathematics)Analytic continuationXML
Interface (computing)Open sourceScaling (geometry)Mechanism designBitBuildingData storage deviceMathematical analysisDimensional analysisFatou-MengeJSON
Interface (computing)Point cloudPoint (geometry)File formatSoftware developerProjective planeMathematical optimizationArchaeological field survey
PredictionMaizeSatelliteSoftware developerAutomationPredictabilityShape (magazine)Point cloudEndliche ModelltheorieWater vaporOrder (biology)Projective planeRandom matrixDependent and independent variablesState of matterScaling (geometry)Process (computing)Open sourceCodeComputer animation
Open sourceDot productForestAreaDependent and independent variablesSource codePrice indexMeasurementProjective planeSequelRight angleServer (computing)Game theoryComputer animation
Server (computing)Group actionQuery languageMathematical analysisComputer networkQuery languageServer (computing)Level (video gaming)Element (mathematics)Software developerGroup actionQuicksortOpen setObservational studyComputer animation
Electric currentSource codeAnalogyObservational studyFocus (optics)Open setField (computer science)QuicksortSoftware developerGroup actionWebsiteRight angleJSON
WhiteboardGraphics processing unitConnectivity (graph theory)Grass (card game)Green's functionProcess (computing)Data storage deviceType theoryInstance (computer science)Software testingPoint cloudIcosahedronCloud computingPoint cloudNeuroinformatikData centerSubsetServer (computing)Right anglePeripheral
Element (mathematics)Event horizonEndliche ModelltheorieUser interfaceInterior (topology)GeometryProcess (computing)Content (media)Service (economics)Level (video gaming)Stack (abstract data type)Dependent and independent variablesTime zoneAreaSource codeMedical imagingField (computer science)Local ringAxiom of choiceInformationData storage deviceComputer networkLoop (music)Dependent and independent variablesBit rateQuicksortEvent horizonWater vaporServer (computing)Open sourcePeripheralSoftwareFrequencyWebsiteMultiplication signLevel (video gaming)Open setNumberField (computer science)Mobile WebNeuroinformatikConnected spaceVisualization (computer graphics)LaptopMedical imagingData storage deviceState of matterCAN busLocal ringXMLUMLComputer animation
Mathematical analysisTexture mappingEnumerated typeField (computer science)Local ringLevel (video gaming)Visualization (computer graphics)Open setServer (computing)Medical imagingLaptopInterface (computing)Goodness of fitPrototypeSoftware development kitComputer animation
Telephone number mappingSpacetimeArtificial intelligenceSource codeTexture mappingGroup actionInterface (computing)Image resolutionBoundary value problemEvent horizonPoint cloudQuicksortFigurate numberRight angleSpacetimeWeightComputer animation
MereologySatelliteImage resolutionDifferent (Kate Ryan album)QuicksortContext awarenessDependent and independent variablesSpacetimeComputer animation
SpacetimeRoutingMultiplication signDependent and independent variablesEndliche ModelltheorieQuicksortContext awarenessBuildingAdditionMultiplication signRoutingMedical imagingSatelliteBridging (networking)MetreSpacetimeSoftwareComputer animation
Element (mathematics)MereologyPoint (geometry)Interface (computing)Right angleField (computer science)PeripheralDependent and independent variablesQuicksort
Point cloudPrototypeQuicksortComputer programmingPoint cloudJSONComputer animation
Transcript: English(auto-generated)
Thank you. All right. Hello, everyone. So I'm Joe Flasher. I'm the Open Geospatial Data Lead for Amazon Web Services, and I'm going to talk to you a little bit about some of the stuff that we are doing, but then mostly about what our customers are doing
related to disaster response. So first, I'd actually like to start and propose a name change, because I realized some of what I'm going to be talking about is related to software, but it's not just software. It's also going to be open data, right? So I thought we should call it, like, free and open source software and data for Geospatial.
That sounds terrible. So then I thought this is a really great idea. It was after, like, the other night. So we'll see. I thought that we could just add another letter. So maybe, like, free and reusable open source software and data for Geospatial. I really like this, because then we get to have a cool mascot, right? Apologies. I had to
find something that's labeled for commercial reuse, as we're a commercial entity. So that's why you got this. And I would just like to propose that we do this in time for Calgary, because it seems like it would be appropriate. So anyways, I don't know how to actually make this petition, but I'm all into frosty 4G. Okay. So like I said, I work for Amazon
web services. I work in the open data team. Part of my job is to oversee our public data set program, in particular, the Geospatial data sets. If you've seen me talk before, you've almost definitely seen me talk about this. We work with a lot of awesome customers to make data available in the cloud, and we try to do this in smart ways that make it easy
to analyze in the cloud. So this is as opposed to a lot of traditional ways of making data available, which requires you to download massive amounts of data to your own computer, store it yourself, and work with some not necessarily friendly formats. We try to change that, and we work with a lot of our customers to do this. Specifically for Geospatial,
for the last couple of years now, we've had something called Earth on AWS, which is basically a home for Geospatial-related things on AWS. So if you go to this page, you will see a link out to a whole bunch of data sets that we have available that you can use and access, as well as stories from our customers talking about how they're performing their
Geospatial workloads in the cloud. Because it's my firm belief that it's not just enough to make data available, you need to help people figure out how to use that data, or else it's just kind of pointless. So if you've seen me talk before, or hopefully you know that we make a lot of this data available, if you don't, please go there, check it out. But what
you might not know is that AWS also works with a lot of our customers around disaster response-related activities. So we do this in a couple different ways. We actually do deploy into disaster response scenarios. We will send people out into the field. I myself have deployed in this capacity. I was down in North Carolina after
Hurricane Michael in the U.S. And we work with partners on the ground, like ITDRC, to actually rebuild connectivity in communities. We also work with groups like Humanitarian OpenStreetMap team to use our rather large employee base to do mapathons and help map areas around sort of the hot tasks, right, just like
hopefully you all do. And then we also work with a lot of groups to try to incentivize and support the building of applications for good. And so I'm going to talk a little bit about that more in a moment, because that's really leveraging our customers in that capacity. So I sort of wanted to combine these two things, the sort
of open data public data set piece that I see on a regular basis, as well as the disaster response piece that we spun up around two years ago. And so I wanted a little sort of guiding sentence for what I just sort of wanted to think about. And so it was to provide meaningful actionable information as quickly as possible to respond to
disaster situations. And again, AWS, we're a cloud provider, we're an infrastructure company, so I'm trying to think of how can we leverage our infrastructure knowledge and what our customers are doing to sort of help in these sorts of scenarios. So the first thing we can bring to bear is the publicly available data that we make available on AWS. I had a link for this earlier, but if you didn't see it,
the best place you can find all this publicly available data is the registry of open data at AWS. It's just registry.opendata.aws. And here I have the link that if you just want to see data sets that are tagged for disaster response, you can do so. So these are data sets. A lot of it's the satellite imagery that
folks have been talking about, like Landsat, Sentinel-2 data, but also there's like elevation data, so if you're trying to figure out which way water is going to flow, or like people often walk downhill, right? So if you want to figure out which way somebody's going to walk, if they're like lost out in the woods somewhere, like you can use this to figure out which way they're going to go.
We also have, there's a really cool group called Grio. They have the open earthquake early warning network, I think, but it's seismic sensors, real-time seismic sensors that are putting data out through AWS in Chile and Mexico, I believe. So there's a whole bunch of varied variety of data sets there to look
at. And just sort of, I just can't not include this slide because I think it's so awesome. When you think about access to data for disaster response, one of the main things that you need to think about is latency, right? If you're getting access to imagery, but it's hours and hours delayed, that's less helpful to you. So we heard from a lot of our customers who own satellites
that they weren't happy with the latency that they were seeing and how long it was taking them to get information to their customers. And so we said, okay, we will build a ground station network for you. So last year we launched AWS ground station to help customers who have satellites get their data into the cloud much more quickly. So Maxar is a launch company for this, Digital
Globe, you might know them as previously. So it took them, previously it took them about an hour to get the data from their satellite capture into the cloud, which is where they're doing all their processing and distribution of their data. And when they started using ground station in their tests, that went down to a minute. That's huge in a disaster response scenario, right? Like you can get data to people on the ground
much more quickly. So I just love including this slide. So okay, we've got a lot of data, but that's not it, right? You need to be able to index it and find it. And so I'm not going to belabor this point too much because hopefully you've already seen a bunch of the stack talks. But we're trying to do a better job of making the data that we
make available have associated stack metadata alongside it, right? So if you do that then, you can then build indexes on top of it. And so this is work that LMN 84 has done. So Matt Hanson's here from LMN 84. But this is built on top of SAT API. It's run by LMN 84. And so it is an API that you
can query all the geospatial data sets that are made publicly available in AWS that have associated stack metadata. So this isn't everything because we haven't gotten stack metadata in for everything yet. But as we add more stack metadata to those public data sets, you'll see them show up here. So if you're building any applications
or just want to try this out for yourself, you can go here. Like I said, this is all managed by LMN 84. And it will be a good representation of all the geospatial data that's on AWS. So we've got data. We have some sort of system that you can use to see what data's there. So what are people building with this? Or what can you do with data that's made available in the cloud smartly?
So there's a whole bunch of interfaces that you can build on top of this, right? And so I'm just going to go through some of these quickly. This is work that was done by Vincent years ago. He won't give me something new to put in here. I did ask him the other day, but he said to still use this. So this is a slippy map like all of you have seen. And this isn't new anymore. But what this is doing is actually using serverless
technologies to do map tiling, right? So the reason why this is important is twofold. There's no underlying server running here, right? And so Vincent's here in the audience, right? So like Vincent doesn't have to pay to have a server running continuously. So if nobody's making any requests to this mapping interface, Vincent doesn't have to pay to keep a server running if nobody's using it, right?
The other piece that's important in the disaster response context is that disaster response workloads are very spiky, right? Like you have nothing going on, and then all of a sudden the flood comes in, and you have a lot of people that are very interested in what you have. The serverless technologies are designed to scale up. So you have zero cost when nobody's using it, and
they immediately scale up as they get used, right? Which is a very cool property that you would like to see in disaster response scenarios. So this is something similar. This is work done by Synergize, powered by Sentinel Hub. And so it's the same sort of thing. It's using serverless in the backend, but this is showing it in an analytical context instead of a visualization context.
But it's the same idea. There's a trigger being made, function is spinning up very quickly, grabs the data it needs and return it in some context. This is just showing it for analysis rather than visualization. This is work that was done by David Bittner before Phosphogy NA this year. And he actually put Map Server inside a Lambda function.
So that's the serverless technology I was talking about. So again, you don't need something running continuously. If nobody's using it, it'll spin up on demand. Pangeo has been talked about a little bit. I think Julia might have mentioned it the other day in one of the keynotes. Pangeo is an open source community building large scale distributed
analysis tools. But the ability of Pangeo to work efficiently is predicated on the fact that it has access to optimized data sitting in an efficient storage mechanism. This is work that Hobu did. I think Connor is out here somewhere. I have not met him before, so he might actually be sitting here. I don't know what he looks like.
But this is work that was done on top of the USGS Lidar data. I don't know if you can read the point in the top there, but that is 11 billion points that were analyzed and put together. This is a whole bunch of Lidar point cloud data over the US. The cool thing about this is Hobu did this for USGS. USGS themselves, so this is our US Geological Survey.
USGS themselves said that they had never been able to see all their data in a cohesive manner until this work was done, right? And so this is putting Lidar into a cloud optimized format, which is pretty cool, since our government paid for it all. So it was nice for them to be able to see it. So a couple other projects just to call out, this is development seed. They did hurricane intensity predictions.
This is on top of our US Meteorological Agency data. This is GOES data. These are geostationary satellites. This data is available on AWS. But then what they did was do a deep learning model that would basically automate hurricane intensity predictions because they can look at, there's a technique where you can look at clouds, like the shape of clouds, and
you can figure out the hurricane intensity. And this is work that was actually, I think, done by hand previously, but it would take on the order of six hours. And they were able to automate this and get it down to about 15 minutes, which is, you know, that seems pretty awesome. Blue Dot Observatory. Anjay, I think, maybe gave a workshop earlier on in the week.
But this is work that was done by Synergize. So they were using Sentinel-2 data. They were analyzing water bodies around the globe. This is actually in Cape Town when one of their major water bodies was running out of water when they had a water crisis previously. And so the cool thing here is, again,
all this data is made available in the cloud. This is Sentinel-2 data. But they are able to analyze this at scale, but they're able to do this for a few dollars a month, which is fantastic, right? They analyze water bodies from around the globe, and they can do so for a few dollars a month, which is pretty cool. And I believe, Anjay's not here in the crowd, I don't think, but
I believe this code's all open source, so it's the Blue Dot Water Observatory. And so all of these can be triggers. If you're trying to like set up a disaster response pipeline, right? Like these could all be triggers. So if you start to notice water bodies deteriorating, like that's potentially an indication of drought conditions, things like that. So you can use these as triggers.
This is another project. The data's available on AWS. This is OpenEQ, it's all open source community and open source project. And so this is an aggregation of air quality measurements from around the world. I think it's something like 75 countries, 120 different sources. If I was more on top of my game, I would have taken a picture of like an actual forest fire happening somewhere. But I just took this the other day.
And so you can see that one red dot. That one red dot is high PM10 particulate matter in the air. So this is potentially a trigger of like a fire going on in the area, right? Or you could use this after the fact and see what the exposure is. So when you do look at this over like a populated area like California, when there's wildfires, you will see a lot of dots light up red.
Those are all sensors on the ground. We also make OSM data available, so you can easily query that, right? If you're looking, so you can write something like a SQL query. This is work done by Seth Fitzsimmons. You can make a simple SQL query. And again, you don't need to run any servers to do any of this. So I sort of showed you a bunch of these examples, but
we wanted to know what people who were deploying actually wanted. So we worked with LMN 84 to do a user needs studies. And we talked to groups like America on Red Cross, Humanitarian Open Street Map, Doctors Without Borders. As well as tool developers like Open Drone Map, Mapillary, to see what we could sort of try to put together for people on the ground.
So I would recommend you go there to that site on the bottom right there, and you can see what sort of came out of all that. But for the rest of this talk, I'm just gonna focus on the one piece, which is people wanted access to data in the field. And being able to take compute into the field with them would likely be very helpful. And so the main tool that we have to offer there is something called
the Snowball Edge. This is a physically hardened device that you can actually pick up and take with you. So a lot of people think of the cloud as just something that exists. Like I think it was Yvonne the other day, it says other people's computers, which is true. But those computers don't need to be sitting somewhere far away, you can actually take one with you. So this Snowball Edge actually gives you access to the same cloud services,
some subset of the same cloud services. And you can actually physically take that with you. Like you can see there's an airline shipping label on there, right? Like you can check it on an airline. And it gives you access to cloud services. So this was actually made as an import-export device. A cool little example of this is USGS recently, when there were volcanoes in Hawaii.
They were worried that their data center was gonna get overtaken by the volcano. So they were gonna lose all their data. So we actually shipped one of the, well, we shipped a snowball to them, right? This is a physical device, so we shipped it to them. They stuck all their data on it, and it's got a little Kindle. So you just flip the front of it, and it changes the shipping label. It gets sent back to us, and all of the data was now out of their physical like data center that was located in the lava flow path.
And it got moved into the cloud. And I also wanted to include a fun GIF, so. So okay, I've got five minutes left. So I'm gonna go through this pretty quickly, I apologize. But disaster response data pipeline, what would this look like, right? So you can think of event triggers, right? So we already saw some of these. If you're getting earthquake notifications, wildfires, you see smoke,
you see sort of water disappearing from water bodies, you can trigger on that. So you can start gathering up tools, right? So you pull open street map data like we saw. You gather Sentinel-2, Landsat data. If you have commercial data available, you can gather that as well. There are a number of open source packages to allow you to do this.
Then you can start provisioning servers, right? So you can use things like serverless, or you can actually provision like always on servers for the period of time that you need it. You can start putting together static content websites that you know people are gonna start accessing, and you can start caching them ahead of time, since you know you're gonna get a lot of demand. And then you can start putting this data onto the physical device.
And you can ship that device into the field, so you can walk into the field with something that you have locally that requires no external connectivity. And it will give you all your base map, it will give you all your pre-disaster imagery. And then you can start doing things in the field. You can start collecting drone imagery that can go right onto the device. You can start using things like field papers.
Any way that you can sort of collect data locally, you can put that back onto the device. And that device gives you compute on the edge, storage, and it'll run a network. You can also just stick a network in front of it. And so any of your mobile devices or computers can access that. So a couple of examples of this. This is work we did with Seth about two years ago, Seth Fitzsimmons. So this is Open Aerial Map running on the Snowball Edge for
local data visualization. This is an example of using QGIS on a laptop using the device as an image server. You can also use something like Open Map Kit and PASM running on the device. So this is all things that we've already prototyped running on the device. LM84 has some interfaces that they're using to try and
bring all these pieces together. And if you're collecting data locally, we also work with groups to sort of figure out how best to push the boundaries of AI and ML on top of geospatial data. So if you're doing something like drone collects, or if you're getting high resolution imagery after the event, what can you do sort of locally or in the cloud to best analyze the data, right?
And so if you saw Ryan talk the other day or Jake and Nick are here as well, this is work that we're doing through SpaceNet. And so we're working with SpaceNet to push the boundaries of AI and ML on top of geospatial data and sort of try to help the community get there as well.
And so there's high resolution imagery as well as labels for a bunch of different cities around the world. So there's two challenges I just wanna call out real quick. SpaceNet 4 and SpaceNet 5, cuz I think they're both very important for the disaster response context. SpaceNet 4 took off-nader imagery.
So the question was, how off-nader can you get and have your model still perform? The reason why that's important in a disaster response context, if you want the quickest imagery, you're not necessarily gonna have a satellite directly overhead. So you wanna know how far off-nader will your model still work. And so how quickly can you get those sort of first look images and have your model still work? SpaceNet 5 is cool, because in addition to doing road extraction and
building actual routable road networks, it's looking at travel times. This is extremely important, cuz if you have a bridge that no longer exists and you need to re-update your routing locally, you want to know how fast you're gonna be able to get any sort of response vehicles around to any given place, right?
So being able to do this on the fly with updated imagery that's coming in post-disaster is pretty cool. So SpaceNet 5 is just starting here in the very new future. So if this is interesting to you, please check it out. And just remember this is all part of this sort of what would it look like to have a disaster response pipeline that you could spin up at any point, right?
So you've got your trigger, you gather your data, you build the interfaces, you deploy it to a physical device, and then you ship it out into the field with someone. So with that, I'm gonna stop, but I just wanna say if any of this seems interesting to you, any of the work that any of those folks are doing, please let me know. We have a credits program.
If anyone's looking to prototype any workflows on top of geospatial data in the cloud, please just let me know or go to the website. We are always looking to sort of support those efforts. And with that, I will stop. Thank you.