Scaling for NYC while Tracking Plows
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 188 | |
| Autor | ||
| Lizenz | CC-Namensnennung 3.0 Deutschland: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/31634 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache | ||
| Produzent | ||
| Produktionsjahr | 2014 | |
| Produktionsort | Portland, Oregon, United States of America |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
| |
| Schlagwörter |
FOSS4G 2014 Portland135 / 188
7
10
14
15
16
23
24
25
28
29
33
37
39
40
43
45
46
48
50
56
64
65
69
72
74
82
89
91
98
102
107
111
114
118
128
131
132
135
138
141
143
147
149
150
157
158
161
164
165
166
173
174
175
179
185
00:00
CodeDatenverwaltungInformationPerspektiveSelbst organisierendes SystemSoftwareGebäude <Mathematik>Produkt <Mathematik>InzidenzalgebraMAPBildschirmfensterGrenzschichtablösungEntscheidungstheorieAggregatzustandBitErwartungswertGeradeGruppenoperationPhysikalisches SystemProjektive EbeneQuick-SortAutomatische HandlungsplanungNichtlinearer OperatorNotebook-ComputerVarietät <Mathematik>SchnittmengeArithmetische FolgeKartesische KoordinatenBrowserVollständiger VerbandClientBitrateOpen SourceSchreib-Lese-KopfObjekt <Kategorie>SondierungMultiplikationsoperatorFigurierte ZahlSpezifisches VolumenSoftwareentwicklerBildgebendes VerfahrenFormale SpracheTelekommunikationVerfügbarkeitTypentheorieProfil <Aerodynamik>DialektTotal <Mathematik>Globale OptimierungAnalytische FortsetzungAbfrageDeltafunktionProzess <Informatik>Perfekte GruppeCoxeter-GruppeSoundverarbeitungEreignishorizontPunktwolkeSelbstrepräsentationMapping <Computergraphik>Rechter WinkelDienst <Informatik>SmartphoneDemoszene <Programmierung>
07:29
InformationSpieltheorieGebäude <Mathematik>TouchscreenRechter WinkelBildschirmsymbolComputeranimation
08:09
EntscheidungstheorieFunktionalIndexberechnungMultiplikationSchedulingVisualisierungBildverstehenRoutingArithmetische FolgeKartesische KoordinatenVollständiger VerbandGraphfärbungMultiplikationsoperatorSpezifisches VolumenCodeMenütechnikPerfekte GruppeNabel <Mathematik>Wort <Informatik>PunktwolkeWeg <Topologie>
09:40
AlgorithmusPlotterIterationVektorraumEntscheidungstheorieArithmetisches MittelBitGeradeGruppenoperationMultiplikationTermTreiber <Programm>VisualisierungQuick-SortZeitrichtungPunktInformationsspeicherungKartesische KoordinatenRichtungIdentifizierbarkeitMultiplikationsoperatorRechter WinkelVolumenvisualisierungGamecontrollerFunktionalFehlermeldungZeitstempelComputeranimation
12:55
CodeProdukt <Mathematik>Inhalt <Mathematik>SpeicherabzugReelle ZahlVerzweigendes ProgrammVarietät <Mathematik>Open SourceWeb SiteHumanoider RoboterDesign by ContractGlobale OptimierungQuelle <Physik>SummengleichungRechter WinkelCDN-NetzwerkEins
14:22
DatenbankDatenverwaltungRechenzentrumSoftwareMAPElementargeometrieGlobale OptimierungGeradeInhalt <Mathematik>RechenschieberTabelleZählenDatenflussSerielle SchnittstelleAdressraumElektronische PublikationPunktwolkeSichtenkonzeptZeitstempelMultiplikationsoperatorTesselationRechter WinkelDienst <Informatik>VolumenvisualisierungSpezifisches VolumenCDN-NetzwerkTypentheorieBitZahlenbereichZellularer AutomatQuick-SortServerCoxeter-GruppeInformationsspeicherungComputeranimation
16:14
DatenbankTelekommunikationFrequenzProdukt <Mathematik>MAPHochdruckLokales MinimumPhysikalische TheorieProjektive EbeneRechenwerkSchedulingTermZahlenbereichZusammengesetzte VerteilungProzess <Informatik>Coxeter-GruppeRoutingArithmetische FolgeRahmenproblemPunktwolkeDreiecksfreier GraphMultiplikationsoperatorZweiTesselationRechter WinkelSpezifisches VolumenServer
19:12
AnalysisRauschenPlotterBildschirmmaskeBitGarbentheorieGeradeGlättungKette <Mathematik>SpeicherabzugZahlenbereichQuick-SortFlächeninhaltTeilbarkeitEinflussgrößeÜberlagerung <Mathematik>CASE <Informatik>RoutingPunktQuaderArithmetische FolgeKugelkappeWort <Informatik>RichtungMinimalgradZweiRechter WinkelMechanismus-Design-TheorieEinsWeg <Topologie>Vorlesung/Konferenz
Transkript: Englisch(automatisch erzeugt)
00:00
Just wanted to go over sort of as every project manager really should level set expectations with either the client or the audience, right? So This is not a heavy technical presentation, it's really intended for managers, decision makers, curiosity seekers, those who are interested in plows or those generally interested in what happens in New York City.
00:24
So if that's not you, I won't feel upset if you go and go into one of the other presentations. If not, hopefully you can stick around and I'll entertain and inform you. So with that, let's plow ahead. I wanted the sound effect, but I couldn't figure it out.
00:45
So why should you listen to this presentation? So I'm probably preaching to the choir here if I started spouting out about how we use open source, right? From my perspective working in city government, I thought it was an interesting story to show or demonstrate at least how open source can be used in an otherwise
01:04
very conservative organization. I see in the federal government, there's a fair bit of open source work, but a lot of city and state governments is a real reluctance to it because obviously governments are often risk averse and there's a perception that open source is, you know,
01:21
everybody's managing this code. It's all willy-nilly, you know, and you should be really careful. And there's also a prevailing belief that if you buy shrink-wrap software or commercial software with paid support that you get around it, that you're getting a better product. Well, you obviously need to do the same due diligence in selecting whatever tool set you use, whether it's closed or open source.
01:44
And so New York City is an example of using open source to develop an application that's quite a fairly high profile, gets a lot of use, and that is Plow NYC. So very briefly, I work for the City of New York, Department of Information Technology and Telecommunications.
02:00
It's a mouthful. I bring that up because we're one of 50 some odd agencies within the city. Our mandate is IT services. We don't plow the streets. We provide services. Specifically, I manage the mapping, the GIS group within the city, and
02:21
my role in doing that is I manage a group of about 16 people split between developers, your traditional analysts, systems admin type folks. We manage a lot of the geospatial data for the City of New York. We also have a build and support application. So what we build is what we support. So we're very careful in what we select and how we build things.
02:45
So this situation is what you would have seen looking at your window on December 27, 2010, right? It's been referred to as snowmageddon. It was sort of the perfect storm, right? The mayor and his first deputies were all on vacation in the Bahamas, right?
03:04
The snow rate was very heavy. The accumulation was really a high volume and the temperature was perfect. Quite often the streets are warm and it doesn't start accumulating on the streets very quickly. This was every variable played into a bad storm, really difficult to plow.
03:25
People lost their jobs because of it. Ambulances got stuck, buses got stuck, so on and so forth. So the project. If you fast forward Fast forward about a year, January 2011, Mayor Bloomberg does a weekly radio show and
03:47
the interviewer starts asking the questions about snowmageddon and the 16-point plan that the city put in place. He's like, well, by the way, are you, you know, providing any information to the general public as they might want to know if their streets been plowed or not? He's like, yeah, we'll be doing that and we'll be delivering something this year.
04:02
So I was in a meeting within an hour and found out that yes, we would be developing this application. So we hadn't been. But anyway, we were given, you know, it was January. We had to deliver something by the end of the winter. So it was approximately six weeks to get this out
04:21
of the door. So with that, you know, the first thing we started to do was, you know, think about things, right? So what exactly should we develop here? Because really all we had was a soundbite from a radio show. And so if you want to track plows, you want to see what the progress of plows are like, well, why don't you just look out your window? Because you have visual evidence just looking outside your window. Yes, it's been plowed, right?
04:46
Do you really need to look at a smartphone or look at a web browser on your desktop or laptop to see that streets been plowed or not, right? So we're scratching our heads trying to figure it out. And initially, you know, we came up with some objectives and what we would try to achieve.
05:04
So what we really needed to do was convey the snow operation progress to the residents and visitors of the city of New York. And we say snow operations because they use both plows and spreaders, right? And we needed the ability to handle a large volume of traffic, right? So this is a sort of
05:24
incident where there's a snowstorm, the application gets activated, everybody's gonna come on in, you know, the duration of the snowstorm is maybe a day. They'll leave it on for a bit longer just to show the streets, you know, continue to be plowed. But it's a very short event, a lot of traffic at any one given time, so high availability,
05:43
support a lot of applications, excuse me, users. Really keep it simple and straightforward, right? Convey information in an understandable way to the lay public. And it was really about conveying that information and not really about the technology behind the scenes, right?
06:00
So we wanted to deliver them the minimal viable product, right? Go out with an initial release, meeting most of the undefined requirements, and then go out with future releases and build upon that, right? So it was a total team effort.
06:21
I'm lucky to manage quite a cadre of very good developers. So we did develop this entirely in-house using the tools that we'd already been comfortable with, so learning on the job, new technologies, when you have a very aggressive time line is not advisable. And on existing infrastructure, which was frail and aging,
06:44
but we had to make it work for at least the first winter while we harden things for the next. So essentially the project became two separate efforts, right? You had the mayor's mandate, and then you have a variety of stakeholders, Department of Sanitation plows the streets,
07:00
Office of Emergency Management manages snow emergencies, they handle communication, and then you had City Hall, and then you had my team with a development effort, right? So one taking more of a waterfall, let's meet every other week, pontificate, talk, you know, throw occasional requirements out there, and the development team taking more of an agile approach.
07:21
We had daily scrums. We were actually building what they were trying to formulate in their heads, right? So the first thing we really did was let's take a survey. Let's do let's see what the lay of the land is out there with what some of the other cities are doing, right? Seattle on the top, Chicago there on the right. Not to be critical, but okay,
07:43
so seeing a cute plow icon dance around on your screen doesn't really tell you what's happening out there. It doesn't tell you whether your streets been plowed or not. It's cute and all, might make a better game, but it certainly doesn't convey information. On the top over here, you know, they're plowing more buildings than they are streets, so
08:02
we looked at those and we realized there wasn't really anything out there that really helped inform what it is that we should be doing. So, which brought, you know, some other challenges in place, right? So we need to realize a vision from a person we didn't really have access to. The mayor wasn't gonna sit in any of our meetings.
08:22
He just said we had to do something and we had to hope that we hit the target or weren't too far off the target and then adjust accordingly, right? Multiple stakeholders, the dreaded decision by committee when decision were reached and they focus on minutia like colors and things like that as opposed to like the functionality of the application.
08:43
And what we really wanted to do initially was we would, let's track progress against a scheduled plow route, right? So plows are given a schedule of streets that they need to go out there, a route, and here's what you do for the day. And let's track their progress against what they're expected to cover.
09:00
Well, those plow routes were actually in narratives in WordPerfect documents to just give you in some indication of how old they were, right? And if you read them and tried to map them, there were huge gaps between where they started and where they made left turns and it's, you kind of wonder how the streets actually get plowed if that's the
09:22
narrative that they're following. So quickly we realized that wasn't gonna work, right? So we had to come up with something else. We had the very aggressive schedule and then we needed to handle a large volume of traffic in a, in a, in a fairly short period of time. Right, so we started doing some initial visualizations of the GPS points, right?
09:45
Creating vectors and showing those arrows there or the bearing of the vehicle. And you know the one going down Second Avenue looks pretty good. The one on First Avenue, either that the driver of the plow was drinking that day, either the person drawing that fake line
10:02
had maybe too much to drink that night or there was a multipath error there. So it was safe, but anyway we looked at that and realized that's similar to what Seattle was done and this is not going to really be very helpful, right? So we realized we needed to take an entirely different approach and
10:21
what we decided to do, and this is not going to let me do my transition because it's a PDF, but anyway what we tried to do was or what we did do was we snapped those GPS points to the street segment they should have been plowing and we had some intelligence in there in terms of looking at the directionality of the street,
10:43
the bearing of the plow, looking at previous segments to ensure you weren't automatically making a quick right turn, right? So there was intelligence in that algorithm, snapped it to the nearest street segment, and then we created time buckets. So we're showing here when the street was previously plowed, whether it was plowed in the last hour up to the last 12 to 24 hours.
11:05
And this was a decision in terms of the time bucket that took forever for the committee to make, but anyway this is what we went with. So we get the GPS feed and this is the first iteration, right? And so what happens here is we get the GPS feed,
11:23
we're snapping this to that street segment, which has a unique identifier, every 15 minutes we're pulling that GPS data store and looking at the last time stamp on a street segment and then putting those into the different buckets and then rendering it.
11:42
And I'll give you sort of the technology between how that's being done, but this was probably release 1.5, right? It was there was an earlier release, but this was this still sort of is somewhat emulating the sort of GIS desktop with the concept of layers on the right and a bit more functionality that probably we wanted, right? So this was the sort of
12:03
more recent release, right? It's completely responsive, mobile compliant, right? Get away from having turning layers on and off, right? There's two different things that you can see. Snow vehicle activity, and I took this screenshot recently, so they weren't snow plows in New York City, but it also shows the designation of each streets, whether they're a primary street, secondary, tertiary, meaning do they get plowed first,
12:25
second, or third, or all bets are off. We don't cover your street. So this is the current application. OEM has the controls as well as sanitation to activate. You see on the right-hand side, it'll indicate whether it's active or not.
12:44
Here it was previously at the top, right? And it'll tell you it's current as of what date, time, and then the next time the ETL runs to pull the data and re-render it. So this is how it looks on a mobile
13:00
Android device. So so these are the variety of technologies that we used and one really good anecdote that really is a real plus for open source. If we had went with one of the proprietary solutions that shall go nameless,
13:24
we ran into a defect with one of the core products we were working on, and Boundless provided support to us. We have a support contract with them. We had an engineer on site within 24 hours. We had those defects corrected and posted back to GeoServer within 48 hours, and they gave us a branch of the code which we had deployed within
13:45
three days. So had that had happened with one of those unnamed vendors, that would have been probably months, not days, right? So that's a real plug for open source right there. So these are the variety technologies we use. We use SpringBratch to write the ETL.
14:04
You're probably most familiar with all the other ones. We use Akamai as a content delivery network, so we're caching all the content closest to the end user. So how does it work?
14:21
I had a very simple non-technical graphic that I did to explain this to managers within the city of New York, right? So here's the data flow and here's how things work because they're all like, wow, how does this really work? So there are GPS devices within the plows, which initially started out as being essentially cell phones, and now we're embedding them with full AVL with
14:43
dead reckoning. It goes up to a Zora database, which is it's a partner of Verizon, right, which then goes to NiceWin, which is our internal wireless network. They have a data center where they get all the GPS feeds coming in. At the red line is where we're really
15:03
kind of my team took over, right? We wrote the snap to grid, but we deployed it to those NiceWin servers. Our ETL runs every 15 minutes. It's polling that database, looking at each segment, and the timestamp on every one of those populates
15:21
the ETL runs, populates a table. We have another view on that table that tells us all the segments were there, where the timestamp had changed. We then send GWC requests to render new tiles because everything's tiled.
15:41
Then we serialize those tiles, store them on a disk, and then all that content is then cached again on Akamai. So the first time someone comes in, types for, you know, whatever street address, all those tiles that are being rendered then cache. So within the next 15 minutes, all that content is then coming from the content delivery network and not coming back from our service.
16:05
So it's got multiple levels of caching involved just to ensure that we can handle a sort of volume of traffic that we see. And we do see a fair bit. So a good presentation would be nothing without a certain number of stats,
16:20
right? And you have to read the small print over here on the Asterix by yourself. I'm not going to read that one. But anyway, so theoretical maximum on a 24-hour period, given the number of times that the ETL runs and the number of levels that we have, there's up to 192 million tiles that get regenerated in 24-hour periods.
16:43
So those servers are humming along, right? We get, during a big activation, we get anywhere from one to two million visits. So a lot of traffic's coming in. There's 10,000 kilometers worth of roads. If you think of lane miles, probably four times that for what they have to plow.
17:01
There is 2,800 snow vehicles, mixture of plows and spreaders. So it's a large volume of GPS data. The GPS data is coming in every 10 seconds, but we're pulling the database on a 15-minute cycle. let me see, so lessons learned. Well, some of these were learned on the
17:22
project. A lot of these were just reinforced, right? You always have to listen, listen, listen when you're dealing with a committee. It's certainly a challenge. Communication's always a key, right? And then in terms of more on the technical side, you know, putting out the minimal viable product, especially in an aggressive time frame, is, you know,
17:43
just really key. Get what you can out there and then add to it. It's easier to add than it is to subtract, right? And then, you know, go with what you know, especially in an aggressive schedule like what we had, you know. Don't pick technologies that you haven't used before or you're experimenting. Use what you're comfortable with,
18:03
you know, and hopefully you have processes in place that when an emergency hits, you're able to respond quickly and you have some qualified individuals behind you. And I certainly do. The URL is this, but the Wi-Fi has been flaky down here, so I'm not even going to attempt it.
18:22
But there's really nothing to see, but, you know, if you want to, you know, I'm sure I'll put this presentation out there the next time we activate during a snow emergency. You can check and hopefully things are moving along. And hopefully in the foreseeable future when we do actually have snow routes, we'll be able to track progress,
18:40
you know, where they've been against where they're expected to be. So somebody can come in and say, okay, my street's going to be plowed at 3 p.m., right, as opposed to, wow, it hasn't been done yet, you know, what's going to happen? So with that, that's the end of my presentation. Happy to answer any questions? Yes. Yes.
19:19
I mean, so there is and
19:23
there has been cases where we snap to the wrong street and it's been shown on the public and, you know, actually some of the papers in the city have written about that, right? But yes, there are usually multi, you know, it's being pinged, there are GPS points coming every 10 seconds.
19:43
So usually there's multiple hits on any street segment, right? They're traveling quite slow when they plow the streets, right? And when they're going faster, they're on the highway. So we get usually at least two points for every street segment, but instead of just snapping to the nearest street, we're looking where the plow has previously been, right? So if you're doing 15 miles an hour, you can't, you know, bang a right very quickly, right?
20:04
Because you'll see it straight to a street and then it'll go next. So we hold a certain number of points, we then snap and then we move on. So we're kind of doing a bit of smoothing and sort of, you know, normalizing the data to ensure that we can remove some of the straight points,
20:21
some of the noise, right? The really problematic ones are like when I showed you First Avenue, when you have a whole line that's completely off. That's when we're mistaken, right? And so in that case, that was actually right up last year, we had falsely
20:40
indicated that the adjacent street, York Avenue, had been plowed when in fact First Avenue hadn't. Everybody was complaining, they're telling us the street's been plowed, it hadn't. The fact of the matter is we shouldn't really need to do snap to street because if we had real AVL in these vehicles, that wouldn't be an issue and that's being implemented this year. So it was sort of, it was one of those stopgap measures that needed to be done
21:02
and it lived a bit longer than we expected. Yep. Excuse me? Yeah, so, you know, when something like that happens, we have no microphone to hand out?
21:20
Yeah, sorry. Anyway, the question was how do you defend something like that? For us, it was more we had to defend it internally to the mayor unless he defended it to the press. We had to defend what happened to the mayor and it really became a strong realization, hey, the current technology that we're using, the GPS technology, is insufficient.
21:43
You know, we could spend a lot of money, try to improve the snap to grid and it's never going to work. The best solution is to get a real AVL. So we took one on the chin, but we ultimately, as you can see this year, they're deploying new AVL. Yes, go ahead. Yeah, were the vehicles already equipped with the GPS or did that have to get installed for this project?
22:04
That was, it was being prototyped in a couple of areas and then it was quickly rolled out to all vehicles. It wasn't real AVL. It's more a, you know, a cell phone in a steel box on the dashboard, you know, so it was really low tech.
22:21
And then now that the data is being collected, is it being used to optimize the plowing? That's a really good question. So that's the direction that they're heading in. So the Department of Sanitation is sort of re-engineering how they do things and they're now going to digitize all of the routes. They're going to, hopefully we're tracking progress against those. So they're using it not as just a mechanism for informing the public,
22:43
but also to inform themselves and to improve how they plow things. The thing about, you know, the sanitation department is it's union, it's a union shop. So there's only so much flexibility you have in how you change the way they do things. And they all handle separate sections.
23:01
So there's a degree of autonomy in each one of the sections that they're covered and there's this person out there with a walkie talkie helping to orchestrate things. So yes, there will be improvements made, but you know, sometimes there are limiting factors, unions being one of them. We're going to WordStar.
23:22
And then we might go to Word or we'll just go to Google Docs. Screw Microsoft now. Any last questions? All right. Sorry if I spoke too quickly, but with the technical difficulties, I figured I'd have to make up steam. Thank you.