The Development and Evolution of an open source mapping application within the USG <- Now with More Google Glass
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 188 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/31636 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer | ||
| Production Year | 2014 | |
| Production Place | Portland, Oregon, United States of America |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G 2014 Portland133 / 188
7
10
14
15
16
23
24
25
28
29
33
37
39
40
43
45
46
48
50
56
64
65
69
72
74
82
89
91
98
102
107
111
114
118
128
131
132
135
138
141
143
147
149
150
157
158
161
164
165
166
173
174
175
179
185
00:00
MereologyWeb pagePoint (geometry)Uniform resource locatorService (economics)Degree (graph theory)File viewerElectric generatorAdditionConnected spaceWeb 2.0Symbol tableServer (computing)Event horizonStandard deviationType theoryOperator (mathematics)Incidence algebraField (computer science)Cartesian coordinate systemAlgebraic varietyEmailRule of inferenceRevision controloutputClient (computing)Computer iconView (database)Mobile appComputer fileMultiplication signPoint cloudDescriptive statisticsDebuggerSlide ruleDenial-of-service attackSubject indexingAudio file formatGame controllerCuboidThermodynamisches SystemStudent's t-testFitness functionWordComputer wormFigurate numberRoundness (object)Address spaceTexture mappingPower (physics)WindowState of matterDifferent (Kate Ryan album)State observerRight angleFilm editingGodBuildingGroup actionRegular graphSpacetimeVideoconferencingTouch typingDigital photographyProcedural programmingXML
06:59
Client (computing)Android (robot)Server (computing)Functional (mathematics)TesselationUniform resource locatorBit rateCore dumpRight angleDatabaseComponent-based software engineeringMedical imagingDigital photographySoftwareForm (programming)Array data structureMultiplication signVideoconferencingPoint (geometry)Run-time systemBitThermodynamisches SystemInterface (computing)Texture mappingSpacetimePlug-in (computing)Casting (performing arts)Archaeological field surveyOffice suiteData miningTable (information)Computer fileType theoryWater vaporQuicksortTraffic reportingBuildingWeb 2.0Set (mathematics)State observeroutputService (economics)Front and back endsDebuggerAverageRepresentational state transferIdeal (ethics)BenutzerhandbuchMobile appRevision controlFile formatOverlay-NetzMereologyEvent horizonGroup actionTheoryLevel (video gaming)Drop (liquid)Connected spaceBuffer solutionGoodness of fitInformation and communications technologyComa BerenicesField (computer science)Audio file formatAreaMappingWeightLocal ringCuboidView (database)Different (Kate Ryan album)Stack (abstract data type)Theory of relativity
13:59
Multiplication signOpen sourceAdditionStandard deviationProcess (computing)Digital photographyField (computer science)Uniform resource locatorBoss CorporationPoint (geometry)Augmented realityWeb pageSummierbarkeitMoment (mathematics)WebsiteEvent horizonSharewareQuicksortCycle (graph theory)Right angleCircleSoftwareVideo gameState observerMobile WebBuildingNumberMathematicsOpen setView (database)Disk read-and-write headStatistical dispersionConnected spaceState of matterOcean currentSolitonData storage deviceInstance (computer science)Algebraic varietyArithmetic meanBlogExpected valueAreaSocial classLibrary (computing)Android (robot)Mobile appComputer clusterGeometryEmailDegree (graph theory)Einbettung <Mathematik>SynchronizationMetreSoftware bug
20:58
Right angleClient (computing)VideoconferencingDigital photographyP (complexity)Revision controlWater vaporForcing (mathematics)Texture mappingLaptopField (computer science)State observerPoint (geometry)BitUniform resource locatorWordLoop (music)Data structureHand fanAndroid (robot)Data miningOperator (mathematics)Structural loadLecture/ConferenceComputer animation
24:45
Slide ruleQuicksortComputer animation
33:56
Server (computing)Algebraic varietyHigh availabilityMereologyMultiplication signFigurate numberCycle (graph theory)BitTelecommunicationPresentation of a groupVariety (linguistics)Incidence algebraDecision theoryCartesian coordinate systemRight angleBit rateData managementGroup actionSmartphoneOpen sourceMedical imagingAdditionWindowPerspective (visual)Lattice (order)DialectInformationBuildingType theoryCodeMappingState of matterObject (grammar)Direction (geometry)SoftwareDisk read-and-write headSound effectLaptopSelf-organizationProduct (business)Reading (process)Event horizonPerfect groupVolume (thermodynamics)Profil (magazine)QuicksortExpected valueOperator (mathematics)Process (computing)Web browserSet (mathematics)Dirac delta functionTotal S.A.Level (video gaming)PlanningThermodynamisches SystemTrailClient (computing)CausalityDemosceneArchaeological field survey
41:35
Bit rateInformationGame theoryTouchscreenBuildingComputer iconRight angleWordFunctional (mathematics)MultiplicationTrailPrice indexGraph coloringScheduling (computing)Volume (thermodynamics)RoutingPerfect groupPoint cloudDecision theoryLattice (order)Machine visionCartesian coordinate systemFrequencyMultiplication signVisualization (computer graphics)Computer animation
43:46
Right angleView (database)Multiplication signArithmetic meanArrow of timeBitArray data structurePoint (geometry)Term (mathematics)AlgorithmVisualization (computer graphics)QuicksortGroup actionIdentifiabilityData storage deviceMultiplicationGame controllerDirection (geometry)Line (geometry)Cartesian coordinate systemDevice driverPlotterDecision theoryError messageFunctional (mathematics)TimestampProbability density functionComputer animation
47:01
Content (media)Product (business)WebsiteSpring (hydrology)Open sourceVariety (linguistics)Content delivery networkChemical equationBranch (computer science)1 (number)Mathematical optimizationDesign by contractCodeCore dumpReal numberMobile WebAndroid (robot)
48:28
MiniDiscTable (information)Content (media)TimestampMultiplication signData centerDataflowContent delivery networkLevel (video gaming)Data storage deviceBitView (database)Type theoryTesselationLine (geometry)Server (computing)Service (economics)VolumenvisualisierungData managementCountingSoftwareQuicksortVolume (thermodynamics)Right angleSoftware testingDatabaseAddress spaceComputer fileComputer animation
50:19
Multiplication signTerm (mathematics)Cycle (graph theory)Scheduling (computing)Volume (thermodynamics)Process (computing)Presentation of a groupNumberAlgebraic varietyMaxima and minimaTesselationDatabaseInformation and communications technologyFrequencyMixture modelServer (computing)Letterpress printingLevel (video gaming)Computer fileTheoryRoutingRight angleProduct (business)Group actionExecution unit
53:18
Covering spaceQuicksortDegree (graph theory)Mathematical analysisSmoothingWordCASE <Informatik>Key (cryptography)NumberLine (geometry)Sheaf (mathematics)Direction (geometry)Point (geometry)BitMechanism designCuboid1 (number)Form (programming)Spherical capRight angleMeasurementDivisorRouting2 (number)AreaNoise (electronics)TrailComputer clusterLecture/Conference
Transcript: English(auto-generated)
00:00
So I'm with the Future Experience Group. How's that for a fun name? And Ben's with the GI Mobile Apps team. We have official descriptions of what our teams do. But effectively, we want to try and work on stuff that's useful and innovative. And we don't want research projects that are somebody's personal soapbox that they research
00:23
for three years, and then two people want to use it. So we want to get fieldable technology that can be used fairly immediately. So the history of this app, the initial version of this app is called the UFO app. And it had a really cool icon with a spaceship.
00:41
But then we were told to pull back the spaceship icon, because people might read into that a little bit. So the history of this work came about from urban search and rescue people working in the disaster after Hurricane Irene. And we'll go through all these slides for the PowerPoint
01:01
usage. So one of our better analysts went and supported Irene. And he saw one of the firefighters that was on the urban search and rescue team from Virginia came back. And he presented his field notes to an analyst to make it into a map.
01:21
And this was the field notes. There was three pieces. And bear in mind, these are the little 3 by 5 notepads. The guy spent a pretty in-depth amount of time sketching out the road intersections, making notes, and passing it off for others to use. And I'm sure that took a lot of time.
01:44
If it rains or a strong wind hits, you might have some other issues going on. But our analyst looked at it and kind of just went with, this is stupid. We can do better. We can do a lot better. So cut to two days later, we had stood up
02:01
the first generation of what became called the Urban Search and Rescue Field Observation App. What happened here was it was basically an HTML5 page that was wrapped into a, excuse me, it was an HTML5 web page that you could access from Android, iOS, BlackBerry,
02:21
Windows, or your desktop. It would pull the location from the device, fill it up with an XY. It could reverse geocoded. You could put a date stamp on there. And then down at the bottom, you see the thing that says Type. There's about 30 different incident types, collapse building, animal issue, burning vehicle,
02:43
human remains that you can drop down. You can fill in your team and then type in a description. You hit Submit. And then it writes it to a Esri endpoint. And this was good. This was great. People liked this. And the nice thing, if you all work with the mobile environment,
03:00
is you didn't have to worry about installing anything. It was just a web page you hit. But this technology evolved and people wanted more. They wanted to be able to submit photos, videos, and audio files to the endpoint service. In addition to that, they also
03:20
wanted to have a disconnected ability because you're not necessarily going to have LTE connection after a tornado has hit someplace, or you have flooding, or you have an earthquake scenario. We're going to take a little detour here. What the urban search and rescue teams currently do
03:41
is they have a handheld GPS solution, which is a Garmin. And they have a custom symbol set that they load into the Garmins. And they then extract them at the end of the day. And normally what happens is, having done this in the field, the search and rescue teams will come back and they'll drop 10 Garmins at a desk and tell the analyst to extract it.
04:02
Their older approach, they extracted it, they converted it to a KML file, and then they emailed a ton of KML files around. So if you have a four or five day event, you could potentially have 50 KML files emailed around five different times. You might be dropped off two or three of the email chains. The ability to search and index it
04:22
got spun out of control pretty fast. So what we ended up evolving, what we ended up writing there was basically just a web page importer where, when you extract the data, you get a GPX file, you import the GPX file, it gives you some dropdown combo box
04:41
menus that says what's your team, what type of search is it, who's your squad leader, and then it will load this all into an Esri service. The nice thing with this is, when it gets used, it gets used and everyone has the same end point to hit.
05:02
We do not, will not, shall not, foot stamp to the nth degree dictate what is the viewer that the user wants. We want to provide the service and we want to let them ingest it however they see fit. The urban search and rescue teams love this.
05:21
It's now officially part of their standard operating procedure, so it is written into their rules that they shall use this approach in this application. The first beta view of it came about from the Colorado flood. So these are actually points that the urban search and rescue teams use for the Colorado flooding.
05:42
And then it's coming out, Barty, was the mudslides in Washington state. So the mudslides in Washington state, we had north of 6,000 points in the field. So you had, I think it was 12 teams there doing the points.
06:01
But you had 6,000 points where people could see that there was remains found at this location or there were other search and rescue point events that they needed to find. This approach is also, we work with NGA.
06:21
That's part of the IC. There's a lot of this data that we are not allowed to handle and touch. So all of this front end web page as well as all the data is actually living on FEMA servers in the cloud. So if it's neat and it's all Amazon based,
06:41
so in a big event, in a big disaster, they can ramp it up and put more servers on it as needed. So we'll get back into the UFO app. So the first app was just an HTML5 web page. The second app, we had developed an iOS and an Android client
07:04
to install. We'll talk a little more in depth on the Android app. The Android app was written for the urban search and rescue guys, but it also had an offline component where you could zoom into Centroid, be it on OpenStreetMap, and cache the tiles.
07:23
You could define the buffer. I want a five mile buffer around this area and go down to zoom level 18, and it'll say you're about to grab 452 meg, are you sure you want it? And then you could have the tile on the map. So if you're in an area with no connectivity,
07:40
you could still have a mapping component. When it worked disconnected, you could also take photos, videos, and audio files on the phone. And when you were back in a Wi-Fi or a good connectivity, it would upload them to the service. So this was good. This was great. This was popular. People liked it. The problem with this was we didn't control the stack.
08:04
So the front end of this form was originally designed for the urban search and rescue team. So their drop down boxes were disaster related. And then the back end was an Esri endpoint.
08:21
But we repurposed this app at least six or seven times. And each client wanted a new front end that just were stuff related to what they were dealing with. And then they needed to stand up their own Esri service on the back end. The problem there was our developer needed to reskin the front.
08:42
And then whoever we were doing this for needed to stand up an appropriate Esri service. And if you had Esri tech support, you could do it pretty quickly. But some of the analysts and some of the tech support from other teams and groups were less than ideal. The fastest we ever managed to repurpose an app
09:00
was about three days. On average, it took about two weeks. And we had a couple that went north of four or five weeks. So it was a popular app. But the ability to repurpose it was less than ideal. So that evolved into Mage. And I'll pass this off to Ben since his team wrote it.
09:28
So the other problem was people kept changing how those endpoints worked. We had no control over that. And people changed the format. And so what worked in one version wouldn't work in the next. And nobody would tell us that until we hit it and it didn't.
09:42
So long story short, we said, look, there's a bunch of different people that want this. And you can imagine that there's people even outside Urban Search and Rescue who are coming to us and saying, yeah, yeah, yeah, I want this base capability. But I want the interface a little bit different. I want to report on these types of events and things like that. And what initially started happening
10:01
was we ended up with all these little fiefdoms of people going, yeah, I'll write a custom app for that guy. And I'll write a custom app for that guy. And I'll write a custom app for that guy, which very quickly people started to realize wasn't going to be sustainable. So I came to the table and said, look, there's a basic core functionality here that people want.
10:20
I can build you something that does that and has a customizable UI. So we swapped this all out and started. We really quickly put together a server back end with an API and a front end on it that's built on MongoDB, Node.js, and then using Angular and Leaflet
10:40
for the front end. So it was really quick to stand that up and get it going. And it's got a form where each team that's running this can go in and say, this is what I want mine to look like. And they can define what all that collection looks like. And because we're using Mongo as our back end, there's no schemas. It doesn't care. You build up what kind of data you want to collect,
11:02
and it just happily populates it into that database. So we got rid of that problem. We ended up with our right. We essentially ported over, but started again to make it clean and build native Android and iOS apps. They can pre-cache a bunch of data, so tiles, vectors,
11:21
all pre-cached on the device. So if you're disconnected from the network at any point, you still have access to all of that data that you want. As well as the ability to take, we think that if somebody creates something in the field and they're disconnected, that should be the end-all, be-all of it. They shouldn't have to worry about it again. So if somebody creates an observation and hits Submit on it in the field
11:41
and they're not connected to the network, the app drops it into a local database and hangs onto it, waits till there's a network connection again, and submits it without the user having to do anything. So once you hit Submit, you're done. It's good to go. It will get to the server as long as you have network again at some point. So lets people do all sorts of things.
12:00
Text photos, videos can all be attached. You can submit all that, send it up. It appears on the map. That all goes out to other people who are also connected to that system. So everybody's getting real-time updates. We've used this at a few exercises with the USAR guys and had pretty good success.
12:21
And instead of talking directly to the Esri service, we just wrote a little plug-in on the back end that takes all the data out of this system and will plug it into that ArcGIS online setup if that's what they want to do. Or they can use the web map that we've created here. You can also export, say, files, KMLs, GeoJSON,
12:40
all that kind of stuff. So we're not trying to lock the data into this system that we've built. You can get it out if that's what you want to do. So yeah, I don't care about this slide.
13:02
So right, this is essentially what it looks like on Android. You've got a map. You can see all the stuff. These are all queryable. You can bring them up, see what's happening. You can edit them if you're part of the team. You're available overlays, right? So any cached maps that you have, the app will go out and find them on the device for you and show you, and you can click those on and off.
13:22
If you don't want the map view, we also put in sort of these news feed views so that you can just go see what are the latest observations that are coming in, what are the latest locations for people on my team that are coming in, depending on what you're interested in at any given time. A lot of people are more interested in where is somebody on my team at a given time, right?
13:41
We've heard lots of horror stories in USAR and other places of people sort of disappearing from communications with their home base and people starting to freak out about where they are and where did we last see them. So a lot of people really just like to see, right, where is the rest of my team around me at this time?
14:00
Or where was the last place that we saw somebody if we're having a hard time getting ahold of them? So we got this all up and running. One of the cool things, right, going to the open source point here, some of our, right, so Paul Ramsey, if anybody saw it last year at Nottingham, got up and pointed out, right, being a supporter of open source doesn't just mean
14:21
I use open source, I'm a supporter, right? It means giving back. And so we've built on a lot of awesome open source stuff that other people have contributed, but our guys in the process of doing this were also encouraging them to contribute back. So some of the libraries that we're using under the hood when we've run into bugs or places
14:43
where we saw opportunities for enhancement, our guys are doing pull requests and submitting that stuff back to those projects. And going beyond that, NGA's even been looking to start open sourcing some of the stuff that we have developed. And so this is one of the things that we're gonna be looking at. It is a lengthy, lengthy bureaucratic process
15:03
to make that happen. But we have managed to get a few things out on GitHub now. You can go see NGA's page out there and see what's out there. And we're working to see if we can get through this process with this app too, because we've talked to a number of people who'd like to get their hands on it and use it for other things.
15:21
So we'd like to get it out there where they can just take it and run with it on their own. So going beyond that, right, a bunch of people are coming to us and saying, this is cool, but big thing we've learned besides, the people that we deal with want disconnected. One of the top five questions, if the network's down, what is it gonna do?
15:41
Is this app gonna still work? But then the next thing is, a lot of these guys, they're used to doing things the way they do stuff. And they don't want something that's gonna get in their way or make it feel like it's interfering with their safety. So to the degree that there are opportunities for us to take advantage of some of the wearables and things that are coming out to make them getting these alerts
16:01
and observations fed to them without them necessarily having to hold this phone in their hand and have that distract from what they're doing, we're starting to explore that. So one of the places we started was with Google Glass. I'll let you talk about it. All right, so having just stopped doing an earlier demo where every single connection to the website
16:22
for a live demo didn't work well, we're not going to try and sync up a Glass upload here. Right, but we wrote an APK that we side-loaded into Glass and basically the person could walk, they could click the photo and they hit submit and then they cycle through and pull their event
16:43
and it would submit the point to Glass, pulling the location, embedding the GPS location in the XF header on the photo when it gets submitted and in addition it would write it to the endpoint. And then in addition down here on the lower right, if you could see, there was also an augmented reality view
17:04
so if you had Glass on, while you're turning your head around, you could see that there is a human remains issue, 250 meters in that direction and as you walk it would update and reflect accordingly.
17:21
Now if anybody's ever actually used Glass, it's only $1500 and the battery life on it is an hour and a half while you're actually using stuff. So it's a cool piece of technology, I don't view it as eminently fieldable,
17:40
at least in its current state, so we're in the process of trying to write an instance of this with an Android Wear watch which has a 24, 25 hour battery life, a lot more under the radar than Glass and it also costs $200.
18:01
So if anything happens to that in the field, it's gonna be less of a problem. So that's the Glass tie-in here. We'll pull up a demo, we'll pull up the website here in a second assuming we got decent connections. I'm just gonna take a soapbox moment, probably everybody here already knows this
18:21
but I try to drive this point home when I'm talking inside government circles. So Mike, you'll appreciate this. But right, we really are trying to drive within these circles, right? There's a lot of proprietary stovepipe stuff but that open standards are gonna be key. Getting access to the data sources, right?
18:42
Not having these situations where proprietary things are changing and we're not hearing about it till after the fact. And I don't just mean official, right? I also mean what I call the facto, right? So like I'm all about geo package which is a full on open geospatial consortium, mobile data container, right?
19:01
But I'm also all about geo JSON. So looking at both of those is open standards that we need to be supporting and taking advantage of. And there's some culture change stuff for the government that's having to happen, right? The current expectations that people have from the stuff that they're able to get at home means we've sort of passed this point
19:21
that I think the government got comfortable with and saying, I can give you a piece of crap because I'm the government and you're just gonna use it that we have to, right? We have to keep up and provide things that are just as good if not better if we want people to adopt them and really take advantage of that. And open source software data and standards are all gonna help there.
19:43
And then the most important one and I'm proud to say that Eric's boss's boss actually came up with this and has been shouting it around our agency that we need to embrace change as a constant. And that's not something that anybody who's worked in the government knows that they're good at.
20:03
But so we're trying to drive that in, right? These things are happening way faster. When we're building this mage thing, it's turning around releases, right? And sometimes in a matter of days. We had one customer we were working with who used it during an event and they came back to us afterwards
20:21
and they said it was awesome but we really wish it would do X, Y, and Z. And at their next event four weeks later, it did X, Y, and Z. And they called us up and said, wait, all that stuff's in there? And we said, yeah. And they said, we thought that would be in like two years. And that's the life cycle they're used to living in. So trying to change that,
20:42
trying to take advantage of what mobile brings and make that happen. You gonna try and demo it? So I actually took a... Brave man. Well, I submitted this point before the start. So hopefully it will have uploaded with the very little throughput we had.
21:01
All right, so I installed the mage client on the Android operating system, on the Android phone over there and submitted a point, took a photo, and it was a little bit to the west of Portland. So I'm gonna go into field observations, load.
21:26
Should be one coming up here.
21:51
All right, so this was the point I submitted
22:13
was from the phone. I took a photo of the aforementioned water bottle. You can see it on the left and other people can download those phone,
22:21
those photos, videos, audio clips in the field on a laptop. And then we also have a... We also have a Blue Force tracker ability where people can see where folks' locations are. So I can turn this off and then I can see that I was updated.
22:41
My last location on this updated four hours ago and the map's not redrawing, but we have the lat-long and that's basically akin to this convention center. Yeah, well, hey, I got the photo. Any questions?
23:13
Okay, I'm just repeating it for the other folks. With the updates in the new version of Postgres, have we considered flipping that over
23:20
instead of using MongoDB?
23:44
And just throwing a shout out to Ben's team too. I mean, I'm an Android fan boy above all else and my pet peeve is the battery life, but the GPS on the battery life, the way it pulls, I think you had something that ran like three days once or on a flight back, right? Yeah, so it's not an excessive drain on the battery,
24:03
which is a personal pet peeve of mine.
24:37
Any additional questions?
24:41
All right, well, that's it.
33:58
I had a talk on tracking plows for New York City.
34:02
And with that, I'll take it away. So just wanted to go over sort of, as every project manager really should, level set expectations with either the client or the audience, right? So this is not a heavy technical presentation. It's really intended for managers, decision makers,
34:23
curiosity seekers, those who are interested in plows or those generally interested in what happens in New York City. So if that's not you, I won't feel upset if you go and go into one of the other presentations. If not, hopefully you can stick around then I'll entertain and inform you. So with that, let's plow ahead.
34:45
I wanted the sound effect, but I couldn't figure it out. That's the last one, right? Exactly. So why should you listen to this presentation? So I'm probably preaching to the choir here if I started spouting out about how we use open source. From my perspective, working in city government,
35:02
I thought it was an interesting story to show or demonstrate at least how open source can be used in an otherwise very conservative organization. I see in the federal government, there's a fair bit of open source work, but a lot of city and state governments there's a real reluctance to it.
35:20
Cause obviously governments are often risk averse and there's a perception that open source is, everybody's managing this code. It's all willy-nilly and you should be really careful. And there's also a prevailing belief that if you buy shrink wrap software or commercial software with paid support
35:40
that you get around it that you're getting a better product. Well, you obviously need to do the same due diligence in selecting whatever tools that you use whether it's closed or open source. And so New York City is an example of using open source to develop an application that's quite fairly high profile gets a lot of use and that is PlowNYC.
36:01
So very briefly, I work for the city of New York Department of Information Technology and Telecommunications. It's a mouthful. I bring that up cause we're one of 50 some odd agencies within the city. Our mandate is IT services. We don't plow the streets. We provide services. Specifically, I manage the mapping,
36:21
the GIS group within the city. That is, I manage a group of 16 people split between developers, your traditional analysts, systems admin type folks. We manage a lot of the geospatial data for the city of New York.
36:41
We also have a build and support application. So what we build is what we support. So we're very careful in what we select and how we build things. So this situation is what you would have seen looking at your window on December 27, 2010. It's been referred to as snowmageddon.
37:01
It was sort of the perfect storm. The mayor and his first deputies were all on vacation in the Bahamas. The snow rate was very heavy. The accumulation was really a high volume. And the temperature was perfect cause quite often the streets are warm and it doesn't start accumulating on the streets
37:22
very quickly. This was every variable played into a bad storm, really difficult to plow. People lost their jobs because of it. Ambulances got stuck, buses got stuck, so on and so forth. So the project.
37:40
If you fast forward about a year, January 2011, Mayor Bloomberg does a weekly radio show. And the interviewer starts asking him questions about snowmageddon and the 16-point plan that the city put in place. And he's like, well, by the way, are you providing any information
38:01
to the general public as they might want to know if their streets have been plowed or not? He's like, yeah, we'll be doing that. And we'll be delivering something this year. So I was in a meeting within an hour and found out that, yes, we would be developing this application. So we hadn't been. But anyway, it was January. We had to deliver something by the end of the winter.
38:22
So it was approximately six weeks to get this out of the door. So with that, the first thing we started to do was think about things. So what exactly should we develop here? Because really, all we had was a sound bite from a radio show.
38:41
And so if you want to track plows, you want to see what the progress of plows are like, well, why don't you just look out your window? Because you have visual evidence just looking outside your window. Yes, it's been plowed. Do you really need to look at a smartphone or look at a web browser on your desktop or laptop to see if that street's been plowed or not? So we're scratching our heads trying to figure it out.
39:04
And initially, we came up with some objectives of what we would try to achieve. So what we really needed to do was convey the snow operation progress to the residents and visitors of the city of New York. And we say snow operations because they use both plows and spreaders.
39:22
And we needed the ability to handle a large volume of traffic. So this is a sort of incident where there's a snowstorm. The application gets activated. Everybody is going to come on in. The duration of the snowstorm is maybe a day. They'll leave it on for a bit longer
39:41
just to show the streets continue to be plowed. But it's a very short event, a lot of traffic at any one given time. So high availability, support a lot of applications. Excuse me, users. Really keep it simple and straightforward. Convey information in an understandable way to the lay public.
40:00
And it was really about conveying that information and not really about the technology behind the scenes. So we wanted to deliver the minimal viable product. Go out with an initial release, meeting most of the undefined requirements, and then go out with future releases and build upon that.
40:22
So it was a total team effort. I'm lucky to manage quite a cadre of very good developers. So we developed this entirely in-house. Using the tools that we'd already been comfortable with. So learning on the job, new technologies when you have a very aggressive timeline
40:42
is not advisable. And on existing infrastructure, which was frail and aging. But we had to make it work for at least the first winter while we hardened things for the next. So essentially, the project became two separate efforts. You had the mayor's mandate.
41:01
And then you have a variety of stakeholders. Department of Sanitation plows the streets. Office of Emergency Management manages snow emergencies. They handle communication. And then you had City Hall. And then you had my team with a development effort. So one taking more of a waterfall. Let's meet every other week.
41:20
Pontificate, talk, throw occasional requirements out there. And the development team taking more of an agile approach. We had daily scrums. We were actually building what they were trying to formulate in their heads. So the first thing we really did was let's take a survey. Let's see what the lay of the land is out there with what
41:41
some of the other cities are doing. Seattle on the top, Chicago there on the right. Not to be critical, but OK, so seeing a cute plow icon dance around on your screen doesn't really tell you what's happening out there. It doesn't tell you whether your street's been plowed or not. It's cute and all. Might make a better game, but it certainly
42:00
doesn't convey information. On the top over here, they're plowing more buildings than they are streets. So we looked at those, and we realized there wasn't really anything out there that really helped inform what it is that we should be doing, which brought some other challenges in place.
42:21
So we need to realize a vision from a person we didn't really have access to. The mayor wasn't going to sit in any of our meetings. He just said we had to do something, and we had to hope that we hit the target or weren't too far off the target and then adjust accordingly. Multiple stakeholders, the dreaded decision by committee when decision were reached, and they
42:42
focus on minutia like colors and things like that as opposed to the functionality of the application. And what we really wanted to do initially was let's track progress against a scheduled plow route. So plows are given a schedule of streets that they need to go out there, a route, and here's what you do for the day.
43:02
And let's track their progress against what they're expected to cover. Well, those plow routes were actually in narratives in word perfect documents to just give you some indication of how old they were. And if you read them and tried to map them,
43:21
there were huge gaps between where they started and where they made left turns. And you kind of wonder how the streets actually get plowed if that's the narrative that they're following. So quickly we realized that wasn't gonna work. So we had to come up with something else. We had the very aggressive schedule, and then we needed to handle a large volume of traffic in a fairly short period of time
43:43
is what we expected. So we started doing some initial visualizations of the GPS points, creating vectors and showing those arrows there or the bearing of the vehicle. And the one going down Second Avenue looks pretty good. The one on First Avenue, either that the driver
44:03
of the plow was drinking that day, either the person drawing that fake line had maybe too much to drink that night, or there was a multipath error there. So it was safe. But anyway, we looked at that and realized that's similar to what Seattle has done. And this is not gonna really be very helpful, right?
44:22
So we realized we needed to take an entirely different approach. And what we decided to do, and this is not gonna let me do my transition because it's a PDF. But anyway, what we tried to do was, or what we did do was we snapped those GPS points
44:40
to the street segment they should have been plowing. And we had some intelligence in there in terms of looking at the directionality of the street, the bearing of the plow, looking at previous segments to ensure you weren't automatically making a quick right turn, right? So there was intelligence in that algorithm, snapped it to the nearest street segment. And then we created time buckets.
45:02
So we're showing here when the street was previously plowed, whether it was plowed in the last hour, up to the last 12 to 24 hours. And this was a decision in terms of the time bucket that took forever for the committee to make. But anyway, this is what we went with. So we get the GPS feed.
45:22
And this is, sorry. So this is the first iteration, right? And so what happens here is we get the GPS feed, we're snapping to that street segment, which has a unique identifier. Every 15 minutes we're polling that GPS data store and looking at the last time stamp on a street segment
45:43
and then putting those into the different buckets and then rendering it. And I'll give you sort of the technology between how that's being done. But this was probably release 1.5, right? It was, there was an earlier release, but this still sort of is somewhat emulating
46:01
the sort of GIS desktop with the concept of layers on the right and a bit more functionality that probably we wanted, right? So this was the sort of more recent release, right? It's completely responsive, mobile compliant, right? Get away from having turning layers on and off, right? There's two different things that you can see. Snow vehicle activity, and I took this screenshot recently,
46:22
so they weren't snowplows in New York City. But it also shows the designation of each streets, whether they're a primary street, secondary, tertiary, meaning do they get plowed first, second, or third, or all bets are off, we don't cover your street. So this is the current application.
46:40
OEM has the controls as well as sanitation to activate. You see on the right-hand side, it'll indicate whether it's active or not. Here, it was previously at the top, right? And it'll tell you it's current as of what date, time, and then the next time the ETL runs to pull the data and re-render it.
47:02
So this is how it looks on a mobile Android device. So these are the variety of technologies that we used. And one really good anecdote that really is a real plus for open source.
47:23
If we had went with one of the proprietary solutions that shall go nameless, we ran into a defect with one of the core products we were working on, and Boundless provided support to us. We have a support contract with them. We had an engineer on site within 24 hours. We had those defects corrected
47:42
and posted back to GeoServer within 48 hours, and they gave us a branch of the code, which we had deployed within three days. So had that had happened with one of those unnamed vendors, that would have been probably months, not days, right? So that's a real plug for open source right there.
48:03
So this is the variety of technologies we use. We use SpringBrach to write the ETL. You're probably most familiar with all the other ones. We use Akamai as a content delivery network. So we're caching all the content at closest to the end user.
48:21
So how does it work? Kind of went over that previously, but I had a very simple non-technical graphic that I did to explain this to managers within the city of New York, right? So here's the data flow and here's how things work because they're all like, wow, how does this really work? So there are GPS devices within the plows,
48:42
which initially started out as being essentially cell phones and now we're embedding them with full AVL with dead reckoning. It goes up to a Zora database, which it's a partner of Verizon, right? Which then goes to NiceWin, which is our internal wireless network.
49:02
They have a data center where they get all the GPS feeds coming in and at the red line is where really, kind of my team took over, right? We wrote the snap to grid, but we deployed it to those NiceWin servers. Our ETL runs every 15 minutes. It's polling that database, looking at each segment
49:22
and the timestamp on every one of those populates. The ETL runs, populates a table. We have another view on that table that tells us all the segments were there where the timestamp had changed. We then send GWC requests to render new tiles
49:43
because everything's tiled. Then we serialize those tiles, store them on a disk and then all that content is then cached again on Akamai. So the first time someone comes in, types for whatever street address, all of those tiles that are being rendered then cached.
50:03
So within the next 15 minutes, all that content is then coming from the content delivery network and not coming back from our service. So it's got multiple levels of caching involved just to ensure that we can handle the sort of volume of traffic that we see. And we do see a fair bit.
50:21
So a good presentation would be nothing without a certain number of stats, right? And you have to read the small print over here on the asterisk by yourself. I'm not gonna read that one. But anyway, so theoretical maximum on a 24 hour period given the number of times that the ETL runs and the number of levels that we have,
50:43
there's up to 192 million tiles that get regenerated in 24 hour periods. So those servers are humming along, right? We get, during a big activation, we get anywhere from one to 2 million visits. So a lot of traffic's coming in. There's 10,000 kilometers worth of roads.
51:01
If you think of lane miles, probably four times that for what they have to plow. There's 2,800 snow vehicles, mixture of plows and spreaders. So it's a large volume of GPS data. The GPS data is coming in every 10 seconds, but we're pulling the database on a 15 minute cycle.
51:21
And let me see. So lessons learned. Well, some of these were learned on the project. A lot of these were just reinforced, right? You always have to listen, listen, listen when you're dealing with a committee. It's certainly a challenge. Communications always a key, right?
51:41
And then in terms of more on the technical side, putting out the minimal viable product, especially in an aggressive timeframe is just really key. Get what you can out there and then add to it. It's easier to add than it is to subtract, right? And then, you know, go with what you know,
52:00
especially in an aggressive schedule like what we had, you know, don't pick technologies that you haven't used before or you're experimenting. Use what you're comfortable with, you know, and hopefully you have processes in place that when an emergency hits, you're able to respond quickly and you have some qualified individuals behind you. And I certainly do.
52:22
The URL is this, but the wifi has been flaky down here, so I'm not gonna even attempt it, but there's really nothing to see. But you know, if you want to, you know, I'm sure I'll put this presentation out there the next time we activate during a snow emergency, you can check and hopefully things are moving along.
52:40
And hopefully in the foreseeable future when we do actually have snow routes, we'll be able to track progress, you know, where they've been against where they're expected to be. So somebody can come in and say, okay, my street's gonna be plowed at 3 p.m., right? As opposed to, wow, it hasn't been done yet, you know, what's gonna happen? So with that, that's the end of my presentation.
53:03
Happy to answer any questions? Yes, yes.
53:24
I mean, so there is, and there has been cases where we snap to the wrong street and it's been shown on the public, and you know, actually some of the papers in the city have written about that, right?
53:42
But yes, there are usually multi, you know, it's being pinged, there are GPS points coming every 10 seconds. So usually there's multiple hits on any street segment, right? They're traveling quite slow when they plow the streets, right? And when they're going faster, they're on the highway. So we get usually at least two points for every street segment.
54:00
But instead of just snapping to the nearest street, we're looking where the plow has previously been, right? So if you're doing 15 miles an hour, you can't, you know, bang a right very quickly, right? Because you'll see it straight to a street, and then it'll go next. So we hold a certain number of points, we then snap, and then we move on. So we're kind of doing a bit of smoothing
54:21
and sort of, you know, normalizing the data to ensure that we can remove some of the straight points, some of the noise, right? The really problematic ones are like when I showed you First Avenue, when you have a whole line that's completely off, that's when we're mistaken, right? And so in that case, that was actually right up last year,
54:42
we had falsely indicated that the adjacent street York Avenue had been plowed when in fact First Avenue hadn't. Everybody was complaining. They're telling us the street's been plowed. It hadn't. Fact of the matter is, we shouldn't really need to do snap to street because if we had real AVL in these vehicles, that wouldn't be an issue,
55:01
and that's being implemented this year. So it was sort of, it was one of those stopgap measures that needed to be done, and it lived a bit longer than we expected. Yep. Excuse me? Yeah.
55:22
So, you know, when something like that happens, we have no microphone to hand out? Yeah, sorry. Anyway, the question was, how do you defend something like that? For us, it was more we had to defend it internally to the mayor, unless he defended it to the press. We had to defend what happened to the mayor,
55:40
and it really became a strong realization. Hey, the current technology that we're using, the GPS technology is insufficient. You know, we could spend a lot of money try to improve the snap to grid, and it's never gonna work. The best solution is to get a real AVL. So we took one on the chin, but we ultimately, as you can see this year,
56:00
they're deploying new AVL. Yes, go ahead. Yeah, were the vehicles already equipped with the GPS, or did that have to get installed for this project? That was, it was being prototyped in a couple of areas, and then it was quickly rolled out to all vehicles. It wasn't really a VL, it's more a, you know,
56:21
a cell phone in a steel box on the dashboard. You know, so it was really low tech. And then now that the data's being collected, is it being used to optimize the plowing? Is anyone looking at that? That's a really good question. So that's the direction that they're heading in. So the Department of Sanitation is sort of re-engineering how they do things, and they're cap, they're now going to digitize all of the routes.
56:43
They're going to, hopefully, we're tracking progress against those. So they're using it not as just a mechanism for informing the public, but also to inform themselves, and to improve how they plow things. The thing about, you know, the sanitation department is it's union, it's a union shop, so there's only so much flexibility
57:01
you have in how you change the way they do things. And they all handle separate sections, so there's a degree of autonomy in each one of the sections that they're covered, and there's this person out there with a walkie-talkie, helping to orchestrate things. So yes, there will be improvements made, but you know, sometimes there are limiting factors, unions being one of them.
57:22
Upgrades Word from Word Perfect? We're going to WordStar. And then we might go to Word. Or we'll just go to Google Docs, screw Microsoft now. Any last questions? All right, sorry if I spoke too quickly, but with the technical difficulties, I figured I'd have to make up steam.
57:41
Thank you. Thank you very much.