Simple is Better: An Intro to Event-Driven Serverless Architectures for Faster Disaster Response
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 295 | |
Author | ||
Contributors | ||
License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/43516 (DOI) | |
Publisher | ||
Release Date | ||
Language |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
| |
Keywords |
FOSS4G Bucharest 201968 / 295
15
20
28
32
37
38
39
40
41
42
43
44
46
48
52
54
57
69
72
75
83
85
87
88
101
103
105
106
108
111
114
119
122
123
126
129
130
131
132
137
139
140
141
142
143
144
147
148
149
155
157
159
163
166
170
171
179
189
191
192
193
194
195
196
197
202
207
212
213
214
215
216
231
235
251
252
263
287
00:00
Event horizonServer (computing)ArchitectureOpen setOpen setMathematicsDependent and independent variablesBitWeb serviceMultiplication signOpen sourceSoftwareRight angleLecture/ConferenceMeeting/InterviewComputer animation
01:18
Computer programProcess (computing)Set (mathematics)Information privacyMereologyWeb serviceOpen setPoint cloudComputer programmingFile formatSpacetimeData storage deviceMassComputer animation
01:59
Link (knot theory)Point cloudField (computer science)Theory of relativitySet (mathematics)Web pageWorkloadDependent and independent variablesChannel capacityOnline helpRight angleArea
02:42
Computer programDependent and independent variablesConnectivity (graph theory)Texture mappingBuildingInformationCartesian coordinate systemQuicksortOnline helpBitDependent and independent variablesSet (mathematics)Information privacyOpen setInformationPoint cloudGoodness of fitLevel (video gaming)Different (Kate Ryan album)Channel capacityGroup actionBuildingAreaField (computer science)Basis <Mathematik>Connected spaceCloud computingWater vaporTask (computing)Moment (mathematics)Computer animation
04:16
Dependent and independent variablesOpen setWindows RegistryQuicksortSlide ruleSet (mathematics)Open setWater vaporSatelliteLink (knot theory)Bounded variationVariety (linguistics)Group actionReal-time operating systemWindows RegistryDependent and independent variablesSoftwareHill differential equationSource codeXML
05:24
Directed setService (economics)Workstation <Musikinstrument>DistanceScheduling (computing)SatelliteInformationSoftwareDependent and independent variablesPoint cloudMotion captureProcess (computing)Distribution (mathematics)Right angleSoftware testingComputer animation
06:27
Library catalogCore dumpExtension (kinesiology)Focus (optics)Plane (geometry)SatellitePoint cloudVideoconferencingDigital signalEndliche ModelltheorieTemporal logicPoint (geometry)Process (computing)Subject indexingBuildingComputer animation
06:48
Element (mathematics)Entire functionLibrary catalogAmsterdam Ordnance DatumPrice indexElasticity (physics)Element (mathematics)Information privacyBuildingSummierbarkeitPoint cloudPhysical systemInterface (computing)Subject indexingSet (mathematics)Cartesian coordinate systemQuicksortRepresentation (politics)MetadataComputer animation
07:42
Interface (computing)DemosceneMaxima and minimaInformation managementWechselseitige InformationNewton's law of universal gravitationDenial-of-service attackServer (computing)Category of beingLevel (video gaming)Analytic continuationDependent and independent variablesInterface (computing)MappingContext awarenessWorkloadAreaRight angle
08:47
Web browserInterface (computing)Visualization (computer graphics)QuicksortContext awarenessMathematical analysisSimilarity (geometry)Functional (mathematics)Group actionProcess (computing)Server (computing)
09:12
Interface (computing)Server (computing)Lambda calculusServer (computing)Level (video gaming)BitFunctional (mathematics)Analytic continuationXML
09:29
Interface (computing)Open sourceScaling (geometry)Mechanism designBitBuildingData storage deviceMathematical analysisDimensional analysisFatou-MengeJSON
09:52
Interface (computing)Point cloudPoint (geometry)File formatSoftware developerProjective planeMathematical optimizationArchaeological field survey
10:34
PredictionMaizeSatelliteSoftware developerAutomationPredictabilityShape (magazine)Point cloudEndliche ModelltheorieWater vaporOrder (biology)Projective planeRandom matrixDependent and independent variablesState of matterScaling (geometry)Process (computing)Open sourceCodeComputer animation
12:06
Open sourceDot productForestAreaDependent and independent variablesSource codePrice indexMeasurementProjective planeSequelRight angleServer (computing)Game theoryComputer animation
13:04
Server (computing)Group actionQuery languageMathematical analysisComputer networkQuery languageServer (computing)Level (video gaming)Element (mathematics)Software developerGroup actionQuicksortOpen setObservational studyComputer animation
13:19
Electric currentSource codeAnalogyObservational studyFocus (optics)Open setField (computer science)QuicksortSoftware developerGroup actionWebsiteRight angleJSON
13:57
WhiteboardGraphics processing unitConnectivity (graph theory)Grass (card game)Green's functionProcess (computing)Data storage deviceType theoryInstance (computer science)Software testingPoint cloudIcosahedronCloud computingPoint cloudNeuroinformatikData centerSubsetServer (computing)Right anglePeripheral
15:06
Element (mathematics)Event horizonEndliche ModelltheorieUser interfaceInterior (topology)GeometryProcess (computing)Content (media)Service (economics)Level (video gaming)Stack (abstract data type)Dependent and independent variablesTime zoneAreaSource codeMedical imagingField (computer science)Local ringAxiom of choiceInformationData storage deviceComputer networkLoop (music)Dependent and independent variablesBit rateQuicksortEvent horizonWater vaporServer (computing)Open sourcePeripheralSoftwareFrequencyWebsiteMultiplication signLevel (video gaming)Open setNumberField (computer science)Mobile WebNeuroinformatikConnected spaceVisualization (computer graphics)LaptopMedical imagingData storage deviceState of matterCAN busLocal ringXMLUMLComputer animation
16:35
Mathematical analysisTexture mappingEnumerated typeField (computer science)Local ringLevel (video gaming)Visualization (computer graphics)Open setServer (computing)Medical imagingLaptopInterface (computing)Goodness of fitPrototypeSoftware development kitComputer animation
16:58
Telephone number mappingSpacetimeArtificial intelligenceSource codeTexture mappingGroup actionInterface (computing)Image resolutionBoundary value problemEvent horizonPoint cloudQuicksortFigurate numberRight angleSpacetimeWeightComputer animation
17:33
MereologySatelliteImage resolutionDifferent (Kate Ryan album)QuicksortContext awarenessDependent and independent variablesSpacetimeComputer animation
17:51
SpacetimeRoutingMultiplication signDependent and independent variablesEndliche ModelltheorieQuicksortContext awarenessBuildingAdditionMultiplication signRoutingMedical imagingSatelliteBridging (networking)MetreSpacetimeSoftwareComputer animation
18:51
Element (mathematics)MereologyPoint (geometry)Interface (computing)Right angleField (computer science)PeripheralDependent and independent variablesQuicksort
19:09
Point cloudPrototypeQuicksortComputer programmingPoint cloudJSONComputer animation
Transcript: English(auto-generated)
00:07
Thank you. All right. Hello, everyone. So I'm Joe Flasher. I'm the Open Geospatial Data Lead for Amazon Web Services, and I'm going to talk to you a little bit about some of the stuff that we are doing, but then mostly about what our customers are doing
00:20
related to disaster response. So first, I'd actually like to start and propose a name change, because I realized some of what I'm going to be talking about is related to software, but it's not just software. It's also going to be open data, right? So I thought we should call it, like, free and open source software and data for Geospatial.
00:40
That sounds terrible. So then I thought this is a really great idea. It was after, like, the other night. So we'll see. I thought that we could just add another letter. So maybe, like, free and reusable open source software and data for Geospatial. I really like this, because then we get to have a cool mascot, right? Apologies. I had to
01:01
find something that's labeled for commercial reuse, as we're a commercial entity. So that's why you got this. And I would just like to propose that we do this in time for Calgary, because it seems like it would be appropriate. So anyways, I don't know how to actually make this petition, but I'm all into frosty 4G. Okay. So like I said, I work for Amazon
01:21
web services. I work in the open data team. Part of my job is to oversee our public data set program, in particular, the Geospatial data sets. If you've seen me talk before, you've almost definitely seen me talk about this. We work with a lot of awesome customers to make data available in the cloud, and we try to do this in smart ways that make it easy
01:41
to analyze in the cloud. So this is as opposed to a lot of traditional ways of making data available, which requires you to download massive amounts of data to your own computer, store it yourself, and work with some not necessarily friendly formats. We try to change that, and we work with a lot of our customers to do this. Specifically for Geospatial,
02:01
for the last couple of years now, we've had something called Earth on AWS, which is basically a home for Geospatial-related things on AWS. So if you go to this page, you will see a link out to a whole bunch of data sets that we have available that you can use and access, as well as stories from our customers talking about how they're performing their
02:23
Geospatial workloads in the cloud. Because it's my firm belief that it's not just enough to make data available, you need to help people figure out how to use that data, or else it's just kind of pointless. So if you've seen me talk before, or hopefully you know that we make a lot of this data available, if you don't, please go there, check it out. But what
02:42
you might not know is that AWS also works with a lot of our customers around disaster response-related activities. So we do this in a couple different ways. We actually do deploy into disaster response scenarios. We will send people out into the field. I myself have deployed in this capacity. I was down in North Carolina after
03:02
Hurricane Michael in the U.S. And we work with partners on the ground, like ITDRC, to actually rebuild connectivity in communities. We also work with groups like Humanitarian OpenStreetMap team to use our rather large employee base to do mapathons and help map areas around sort of the hot tasks, right, just like
03:25
hopefully you all do. And then we also work with a lot of groups to try to incentivize and support the building of applications for good. And so I'm going to talk a little bit about that more in a moment, because that's really leveraging our customers in that capacity. So I sort of wanted to combine these two things, the sort
03:44
of open data public data set piece that I see on a regular basis, as well as the disaster response piece that we spun up around two years ago. And so I wanted a little sort of guiding sentence for what I just sort of wanted to think about. And so it was to provide meaningful actionable information as quickly as possible to respond to
04:01
disaster situations. And again, AWS, we're a cloud provider, we're an infrastructure company, so I'm trying to think of how can we leverage our infrastructure knowledge and what our customers are doing to sort of help in these sorts of scenarios. So the first thing we can bring to bear is the publicly available data that we make available on AWS. I had a link for this earlier, but if you didn't see it,
04:25
the best place you can find all this publicly available data is the registry of open data at AWS. It's just registry.opendata.aws. And here I have the link that if you just want to see data sets that are tagged for disaster response, you can do so. So these are data sets. A lot of it's the satellite imagery that
04:42
folks have been talking about, like Landsat, Sentinel-2 data, but also there's like elevation data, so if you're trying to figure out which way water is going to flow, or like people often walk downhill, right? So if you want to figure out which way somebody's going to walk, if they're like lost out in the woods somewhere, like you can use this to figure out which way they're going to go.
05:01
We also have, there's a really cool group called Grio. They have the open earthquake early warning network, I think, but it's seismic sensors, real-time seismic sensors that are putting data out through AWS in Chile and Mexico, I believe. So there's a whole bunch of varied variety of data sets there to look
05:20
at. And just sort of, I just can't not include this slide because I think it's so awesome. When you think about access to data for disaster response, one of the main things that you need to think about is latency, right? If you're getting access to imagery, but it's hours and hours delayed, that's less helpful to you. So we heard from a lot of our customers who own satellites
05:41
that they weren't happy with the latency that they were seeing and how long it was taking them to get information to their customers. And so we said, okay, we will build a ground station network for you. So last year we launched AWS ground station to help customers who have satellites get their data into the cloud much more quickly. So Maxar is a launch company for this, Digital
06:00
Globe, you might know them as previously. So it took them, previously it took them about an hour to get the data from their satellite capture into the cloud, which is where they're doing all their processing and distribution of their data. And when they started using ground station in their tests, that went down to a minute. That's huge in a disaster response scenario, right? Like you can get data to people on the ground
06:23
much more quickly. So I just love including this slide. So okay, we've got a lot of data, but that's not it, right? You need to be able to index it and find it. And so I'm not going to belabor this point too much because hopefully you've already seen a bunch of the stack talks. But we're trying to do a better job of making the data that we
06:41
make available have associated stack metadata alongside it, right? So if you do that then, you can then build indexes on top of it. And so this is work that LMN 84 has done. So Matt Hanson's here from LMN 84. But this is built on top of SAT API. It's run by LMN 84. And so it is an API that you
07:03
can query all the geospatial data sets that are made publicly available in AWS that have associated stack metadata. So this isn't everything because we haven't gotten stack metadata in for everything yet. But as we add more stack metadata to those public data sets, you'll see them show up here. So if you're building any applications
07:21
or just want to try this out for yourself, you can go here. Like I said, this is all managed by LMN 84. And it will be a good representation of all the geospatial data that's on AWS. So we've got data. We have some sort of system that you can use to see what data's there. So what are people building with this? Or what can you do with data that's made available in the cloud smartly?
07:42
So there's a whole bunch of interfaces that you can build on top of this, right? And so I'm just going to go through some of these quickly. This is work that was done by Vincent years ago. He won't give me something new to put in here. I did ask him the other day, but he said to still use this. So this is a slippy map like all of you have seen. And this isn't new anymore. But what this is doing is actually using serverless
08:01
technologies to do map tiling, right? So the reason why this is important is twofold. There's no underlying server running here, right? And so Vincent's here in the audience, right? So like Vincent doesn't have to pay to have a server running continuously. So if nobody's making any requests to this mapping interface, Vincent doesn't have to pay to keep a server running if nobody's using it, right?
08:23
The other piece that's important in the disaster response context is that disaster response workloads are very spiky, right? Like you have nothing going on, and then all of a sudden the flood comes in, and you have a lot of people that are very interested in what you have. The serverless technologies are designed to scale up. So you have zero cost when nobody's using it, and
08:40
they immediately scale up as they get used, right? Which is a very cool property that you would like to see in disaster response scenarios. So this is something similar. This is work done by Synergize, powered by Sentinel Hub. And so it's the same sort of thing. It's using serverless in the backend, but this is showing it in an analytical context instead of a visualization context.
09:01
But it's the same idea. There's a trigger being made, function is spinning up very quickly, grabs the data it needs and return it in some context. This is just showing it for analysis rather than visualization. This is work that was done by David Bittner before Phosphogy NA this year. And he actually put Map Server inside a Lambda function.
09:22
So that's the serverless technology I was talking about. So again, you don't need something running continuously. If nobody's using it, it'll spin up on demand. Pangeo has been talked about a little bit. I think Julia might have mentioned it the other day in one of the keynotes. Pangeo is an open source community building large scale distributed
09:40
analysis tools. But the ability of Pangeo to work efficiently is predicated on the fact that it has access to optimized data sitting in an efficient storage mechanism. This is work that Hobu did. I think Connor is out here somewhere. I have not met him before, so he might actually be sitting here. I don't know what he looks like.
10:01
But this is work that was done on top of the USGS Lidar data. I don't know if you can read the point in the top there, but that is 11 billion points that were analyzed and put together. This is a whole bunch of Lidar point cloud data over the US. The cool thing about this is Hobu did this for USGS. USGS themselves, so this is our US Geological Survey.
10:21
USGS themselves said that they had never been able to see all their data in a cohesive manner until this work was done, right? And so this is putting Lidar into a cloud optimized format, which is pretty cool, since our government paid for it all. So it was nice for them to be able to see it. So a couple other projects just to call out, this is development seed. They did hurricane intensity predictions.
10:40
This is on top of our US Meteorological Agency data. This is GOES data. These are geostationary satellites. This data is available on AWS. But then what they did was do a deep learning model that would basically automate hurricane intensity predictions because they can look at, there's a technique where you can look at clouds, like the shape of clouds, and
11:01
you can figure out the hurricane intensity. And this is work that was actually, I think, done by hand previously, but it would take on the order of six hours. And they were able to automate this and get it down to about 15 minutes, which is, you know, that seems pretty awesome. Blue Dot Observatory. Anjay, I think, maybe gave a workshop earlier on in the week.
11:21
But this is work that was done by Synergize. So they were using Sentinel-2 data. They were analyzing water bodies around the globe. This is actually in Cape Town when one of their major water bodies was running out of water when they had a water crisis previously. And so the cool thing here is, again,
11:41
all this data is made available in the cloud. This is Sentinel-2 data. But they are able to analyze this at scale, but they're able to do this for a few dollars a month, which is fantastic, right? They analyze water bodies from around the globe, and they can do so for a few dollars a month, which is pretty cool. And I believe, Anjay's not here in the crowd, I don't think, but
12:02
I believe this code's all open source, so it's the Blue Dot Water Observatory. And so all of these can be triggers. If you're trying to like set up a disaster response pipeline, right? Like these could all be triggers. So if you start to notice water bodies deteriorating, like that's potentially an indication of drought conditions, things like that. So you can use these as triggers.
12:20
This is another project. The data's available on AWS. This is OpenEQ, it's all open source community and open source project. And so this is an aggregation of air quality measurements from around the world. I think it's something like 75 countries, 120 different sources. If I was more on top of my game, I would have taken a picture of like an actual forest fire happening somewhere. But I just took this the other day.
12:41
And so you can see that one red dot. That one red dot is high PM10 particulate matter in the air. So this is potentially a trigger of like a fire going on in the area, right? Or you could use this after the fact and see what the exposure is. So when you do look at this over like a populated area like California, when there's wildfires, you will see a lot of dots light up red.
13:03
Those are all sensors on the ground. We also make OSM data available, so you can easily query that, right? If you're looking, so you can write something like a SQL query. This is work done by Seth Fitzsimmons. You can make a simple SQL query. And again, you don't need to run any servers to do any of this. So I sort of showed you a bunch of these examples, but
13:22
we wanted to know what people who were deploying actually wanted. So we worked with LMN 84 to do a user needs studies. And we talked to groups like America on Red Cross, Humanitarian Open Street Map, Doctors Without Borders. As well as tool developers like Open Drone Map, Mapillary, to see what we could sort of try to put together for people on the ground.
13:40
So I would recommend you go there to that site on the bottom right there, and you can see what sort of came out of all that. But for the rest of this talk, I'm just gonna focus on the one piece, which is people wanted access to data in the field. And being able to take compute into the field with them would likely be very helpful. And so the main tool that we have to offer there is something called
14:01
the Snowball Edge. This is a physically hardened device that you can actually pick up and take with you. So a lot of people think of the cloud as just something that exists. Like I think it was Yvonne the other day, it says other people's computers, which is true. But those computers don't need to be sitting somewhere far away, you can actually take one with you. So this Snowball Edge actually gives you access to the same cloud services,
14:20
some subset of the same cloud services. And you can actually physically take that with you. Like you can see there's an airline shipping label on there, right? Like you can check it on an airline. And it gives you access to cloud services. So this was actually made as an import-export device. A cool little example of this is USGS recently, when there were volcanoes in Hawaii.
14:40
They were worried that their data center was gonna get overtaken by the volcano. So they were gonna lose all their data. So we actually shipped one of the, well, we shipped a snowball to them, right? This is a physical device, so we shipped it to them. They stuck all their data on it, and it's got a little Kindle. So you just flip the front of it, and it changes the shipping label. It gets sent back to us, and all of the data was now out of their physical like data center that was located in the lava flow path.
15:02
And it got moved into the cloud. And I also wanted to include a fun GIF, so. So okay, I've got five minutes left. So I'm gonna go through this pretty quickly, I apologize. But disaster response data pipeline, what would this look like, right? So you can think of event triggers, right? So we already saw some of these. If you're getting earthquake notifications, wildfires, you see smoke,
15:22
you see sort of water disappearing from water bodies, you can trigger on that. So you can start gathering up tools, right? So you pull open street map data like we saw. You gather Sentinel-2, Landsat data. If you have commercial data available, you can gather that as well. There are a number of open source packages to allow you to do this.
15:41
Then you can start provisioning servers, right? So you can use things like serverless, or you can actually provision like always on servers for the period of time that you need it. You can start putting together static content websites that you know people are gonna start accessing, and you can start caching them ahead of time, since you know you're gonna get a lot of demand. And then you can start putting this data onto the physical device.
16:01
And you can ship that device into the field, so you can walk into the field with something that you have locally that requires no external connectivity. And it will give you all your base map, it will give you all your pre-disaster imagery. And then you can start doing things in the field. You can start collecting drone imagery that can go right onto the device. You can start using things like field papers.
16:21
Any way that you can sort of collect data locally, you can put that back onto the device. And that device gives you compute on the edge, storage, and it'll run a network. You can also just stick a network in front of it. And so any of your mobile devices or computers can access that. So a couple of examples of this. This is work we did with Seth about two years ago, Seth Fitzsimmons. So this is Open Aerial Map running on the Snowball Edge for
16:42
local data visualization. This is an example of using QGIS on a laptop using the device as an image server. You can also use something like Open Map Kit and PASM running on the device. So this is all things that we've already prototyped running on the device. LM84 has some interfaces that they're using to try and
17:01
bring all these pieces together. And if you're collecting data locally, we also work with groups to sort of figure out how best to push the boundaries of AI and ML on top of geospatial data. So if you're doing something like drone collects, or if you're getting high resolution imagery after the event, what can you do sort of locally or in the cloud to best analyze the data, right?
17:25
And so if you saw Ryan talk the other day or Jake and Nick are here as well, this is work that we're doing through SpaceNet. And so we're working with SpaceNet to push the boundaries of AI and ML on top of geospatial data and sort of try to help the community get there as well.
17:42
And so there's high resolution imagery as well as labels for a bunch of different cities around the world. So there's two challenges I just wanna call out real quick. SpaceNet 4 and SpaceNet 5, cuz I think they're both very important for the disaster response context. SpaceNet 4 took off-nader imagery.
18:01
So the question was, how off-nader can you get and have your model still perform? The reason why that's important in a disaster response context, if you want the quickest imagery, you're not necessarily gonna have a satellite directly overhead. So you wanna know how far off-nader will your model still work. And so how quickly can you get those sort of first look images and have your model still work? SpaceNet 5 is cool, because in addition to doing road extraction and
18:23
building actual routable road networks, it's looking at travel times. This is extremely important, cuz if you have a bridge that no longer exists and you need to re-update your routing locally, you want to know how fast you're gonna be able to get any sort of response vehicles around to any given place, right?
18:40
So being able to do this on the fly with updated imagery that's coming in post-disaster is pretty cool. So SpaceNet 5 is just starting here in the very new future. So if this is interesting to you, please check it out. And just remember this is all part of this sort of what would it look like to have a disaster response pipeline that you could spin up at any point, right?
19:02
So you've got your trigger, you gather your data, you build the interfaces, you deploy it to a physical device, and then you ship it out into the field with someone. So with that, I'm gonna stop, but I just wanna say if any of this seems interesting to you, any of the work that any of those folks are doing, please let me know. We have a credits program.
19:20
If anyone's looking to prototype any workflows on top of geospatial data in the cloud, please just let me know or go to the website. We are always looking to sort of support those efforts. And with that, I will stop. Thank you.