Road condition assessment and inspection using deep learning
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 266 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/66531 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Data managementVisualization (computer graphics)SurfaceVideo trackingConditional-access moduleMedical imagingRow (database)Data managementInformationOrder (biology)Product (business)Level (video gaming)VideoconferencingINTEGRALSoftware testingTask (computing)Computer configurationBitMultiplicationPhysical systemResultantRevision controlReal numberPlanningProcess (computing)Connectivity (graph theory)Spectrum (functional analysis)RoutingNeighbourhood (graph theory)outputGraph coloringSurfaceFile viewerCondition numberEndliche ModelltheorieSoftware crackingMultiplication signUniform resource locator2 (number)MappingService (economics)Position operatorAndroid (robot)Software developerMobile appMultilaterationCentralizer and normalizerComputer animationLecture/Conference
05:25
SurfaceMenu (computing)Task (computing)Image registrationMedical imagingDatabaseData managementInformationType theoryWave packetLevel (video gaming)VideoconferencingINTEGRALTask (computing)GeometryState of matterAnalytic continuationBitLine (geometry)Group actionCoordinate systemMereologyMoment (mathematics)Physical systemPolygonExecution unitResultantLocal ringTable (information)AngleCountingNumberQuicksortAreaServer (computing)Process (computing)Asynchronous Transfer ModeRoutingPoint (geometry)Social classCuboidOpen setRaster graphicsElectronic mailing listFrame problemTraffic reportingView (database)File viewerClosed setEndliche ModelltheorieRepresentation (politics)NeuroinformatikMultiplication signDigital photographyMappingComputer clusterGraph coloringComputer animationLecture/Conference
10:40
Task (computing)Computer-generated imageryTask (computing)Data managementMathematicsCategory of beingWell-formed formulaPhysical systemSocial classComputer animation
11:15
SummierbarkeitMedical imagingData managementMathematicsTask (computing)Physical systemCuboidGraph coloringView (database)Uniform resource locatorMobile appComputer animationLecture/Conference
11:59
Medical imagingData managementMathematicsDescriptive statisticsLevel (video gaming)INTEGRALTask (computing)Physical systemReal numberField (computer science)Complete metric spaceElectronic mailing listSurfaceUniform resource locatorService (economics)Right angleComputer animation
12:59
Computer-generated imageryPhysical systemData storage deviceVideoconferencingSign (mathematics)Process (computing)SurfaceData managementMedical imagingDatabaseData managementOrder (biology)Computer architectureNetwork topologyProduct (business)Level (video gaming)VideoconferencingINTEGRALTask (computing)GeometryBitInterpolationMereologyMoment (mathematics)Physical systemSlide ruleSampling (statistics)Virtual machineQuicksortServer (computing)Process (computing)Parallel portSocial classOpen setData storage deviceSpring (hydrology)Latent heatModule (mathematics)Sign (mathematics)Point cloudEndliche ModelltheorieNP-hardDifferent (Kate Ryan album)Software crackingObject (grammar)Front and back endsLoginImage registrationService (economics)Web 2.0Software developerMobile appGreen's functionWeight10 (number)Computer animationLecture/Conference
17:02
XMLUMLComputer animation
Transcript: English(auto-generated)
00:08
Hi everyone, and thank you for being here. I'm going to present today one of the products that we have worked on in the past year, called Road Surface Inspector, which we use in production now in Denmark
00:22
for road condition assessment. I'm going to say a bit later what that actually is, and we are using a lot of deep learning and cool things in the background. As I said, my name is Bogdan. I'm coming from Denmark from a company called IT34, where we do a lot of GIS development in all spectrum of products from CAD to GIS.
00:44
Why have we chose to build a product for road surface inspection? Historically, in order to assess the condition of the road, what that means is finding out if there are potholes, if there are cracks or other damages on the road, you would have to go outside.
01:02
Some of our customers used to drive around and write down, I have here, I have a problem, something needs to be done, and so on. There are options where you take an image and then you have a location next to it and so on. We are trying to automize this workflow as much as possible. Everything becomes a bit faster.
01:21
All the data is managed in one single place so you can see how the condition of the road is. You can plan the work and so on. It's also important to be able to drive around and find out what the condition of the road is multiple times a year, as it's more time efficient. Our system is composed of three main components.
01:43
The Road Surface Inspector app, which is a tool for data collection. It's basically taking videos of the road. Then we have the Road Surface Inspector, which is a web-based solution where all the data
02:03
that's coming from the phone and from the deep learning processing can be seen and visualized from heat maps. I'm going to show a bit later. Now, all the damages that our deep learning system finds on the road, they are just some damages, some information.
02:23
You need to have them in a place centralized. These are the damages which we consider being tasks to be solved. That's why we have built an integration with one of our other products called Task Management, where you can actually see everything. The road have here the tasks. They can be sent to the specific teams.
02:41
We have a team that handles with potholes. We have another team that handles and repairs cracks. Next, I'm going to go a bit into detail on every single component of the system. As I said, Road Surface app is a very simple app. We developed it only for iOS, and that's due to Android
03:05
has a lot of camera models, so it's a bit hard to have good results all the time. There are many, many, many camera models, while iOS, they have this amount of phones, and it's easier for us to say to our customers,
03:21
we would like you to drive with an iPhone, this version or this version. The quality is a bit better from our test. It's a very simple app. You log in, and then you choose for whom you would like to work. We have a lot of municipalities in Denmark, so if you work with different municipalities, you just choose the customer for which you are doing the work.
03:43
Then you are starting to film. You put the phone in the car, in a normal holder, and then it films the road. In the next version that's coming up next week, we're going to have a map so you can see your location, you can see where you have driven, and you can also see your planned route.
04:02
So if you have to drive in a neighborhood, you do it in two days, and then you can see the data from the previous day so you know where to drive. It's as simple as that. You just do your recordings. Data is being uploaded every five seconds. Then as five seconds videos, we are recording a GPS position every second.
04:22
That means that we get back five seconds of video and five GPS positions that will later be processed. Now we have all this data that we capture. We process it, but we also need to see
04:41
what the results are in a nice way. That's why we built what we call Road Surface Inspector Viewer, where you can look at the data and can manage all the routes that you have driven. We call a route from when you start the phone until you stop the phone,
05:00
and you can have multiple routes per day and so on. Here I made some test data in Denmark. You can see on the left side some of the routes that I drove. On the right side, they are just represented with a black color.
05:21
We also have a status. That means what is the processing status of the data. While you are driving, the status is driving. When you stop the recording, the data is already on our servers. We are starting the processing. Then you get the processing as a status, and when it's ready, you can access the actual route and see how the roads are, where you have driven.
05:46
How the results look. At the moment, we have two types of heat maps. One is a heat map in every place where a photo was taken. We just count the number of damages that can be seen from that point.
06:02
It's a very simple way of seeing how many damages of any kind I have from this point of view. This is the heat map that you can see. It's actually just a count on every single point. In the background, you can see some green lines,
06:21
and a bit in the corner is a yellow line. We are doing some computation. We are trying to, just from image and angle and so on, trying to find out how large the damages are. Something that is just specific for Denmark, we have some tables that describe exactly how much a damage should count when you consider the whole road,
06:44
and which classes are more important than other classes. In that way, we can build these nicer colored maps with the state of the road in a municipality or a piece of land. Of course, we can just add any kind of layers and data and so on.
07:07
If you have driven, for example, for a whole week and you are interested to see all the potholes in the area, you can easily make a filtering based on the type of the damage that you want and the time frame that you want.
07:23
You get something like this. You get all the damages in a list with the representation on the map where they were found and also the image so you can see how the damage actually looks. From here in the top corner there, the last button,
07:41
you can click and get a CSV report that has a street name, damage type, when it was recorded, who has recorded all this information available. There is a button on the left corner where it says create tasks.
08:00
I am going to present it a bit later how that works, but every single detection, as we call it, or pothole that has been detected can be considered as a unit of work that needs to be solved. From here it is being sent to our task management system. It is the integration I am going to present in a bit. If you would like, for example,
08:20
I will just go a bit back here to inspect an actual route. I have driven here on this street and I would like to go and see the images and see actually the detections for that area. Then you can just, under action there, you can just go into route, and it is called the route inspection mode, and you see something like this.
08:41
On the right side you have all the points, which is actually the images that were taken from the video and also the heat map on top. Then for each image you have the actual image and then you can see all the detections with the classes.
09:01
Technically for this part we are using, on the right side it is Mapbox. In the middle it is OpenLayers, and the image is considered a raster from 00 coordinate, and all the bounding boxes that are coming from our model are saved in the PostGIS database as a geometry. In the local coordinate system close to 00,
09:20
that means that we can draw them on the map, we can modify the polygons, and we can put labels and so on. We made a viewer based on OpenLayers. We can have buttons and so on. The damages that you can see in the image are also visible on the left side. Here the user has the possibility to inspect,
09:42
are these okay? Maybe the model made some mistakes, we are improving the model as we speak. The user has the possibility to delete some of the damages. In the moment you delete the damage, of course, the heat map will also reflect that and the color will change, the count will change. More importantly, the user has the possibility
10:03
to draw another bounding box. Maybe a pothole was missed, so you can just draw another bounding box, you can create a task and so on. What we do with the bounding box is that the user have driven manually, we are saving them in the database as damages or annotations that have not been
10:21
in the training until now. Then we do some sort of continuous training, continuous deployment if you want. Then it's a sort of crowdsourcing annotation as we keep in the database everything that someone has chosen to annotate.
10:43
Go here, they have a button there up, it's an active create task, the blue one. When you create a task, you just do the formula, you say, I have this annotation, I'd like to create a task so someone can do something with it. You can choose a priority, and then you can either leave the same class,
11:03
the pothole maybe was identified wrongly, it should be a crack instead. Then you can also change the category and then create a task in our task management system. As soon as you have created a task or sent it to our task management system,
11:21
the color of the bounding box changes, you can no longer edit it because it's already sent to be solved. You can see here in the image. Then a quick view of the task management system is something we have developed ourselves. It's initially built only for JS tasks.
11:43
All our municipalities are using it for a lot of tasks. We have apps that can send tasks based on your location, and then the municipalities are handling the tasks. All the employees have a user, they log in, they receive their task, there's a possibility. I can show you here how it looks.
12:02
The location on the map, you have a task list on the right side up, and then the details of the task. You can have description, you can have images, comments back and forth with the people in the field, status change, integration with other systems. It's a complete task management system that we use. It's not only for JS,
12:21
it can also be used for IT support, tickets, or anything like that. Tasks from road service inspector are coming in this system, and they can be sent further to be solved. A task has a status when it comes to the system.
12:41
It's registered. When we send it to task management system, it's becoming assigned to someone. When that someone has finished the task, the status changes to completed. As soon as status changes to completed to a task that came from road service inspector,
13:00
the status also changes in this system. That's becoming green. That means that have been solved. It's a full integration with the task management system where you can see all the damage that you have and you can plan accordingly when and who. A bit of system architecture.
13:23
For the web part, we are using Vue.js, Mapbox, and OpenLayers for the map part. Backend for the map part, at the moment we are using Node.js. We are planning to switch to the net.
13:43
Then for data storage, it's Postgres, PostJS, and all the data is on premise. Most of our customers, they don't trust the cloud, so we keep all our data with us. If someone wants to have their data available in their own GIS system, then we use GeoServer to publish it as a VMS or VFS service.
14:06
Road service inspector app has its own backend and sends data to something called GeoProcessor, which is our internal job queuing system, which we have developed internally because we have so much processing to do every single day.
14:22
The data is being sent to servers on premise and Postgres. GeoProcessor, I don't have a lot of slides about it, but it's an end-to-end system for publishing pipelines which are made out of jobs in a specific order, parallelization, choosing on which machine something should go,
14:43
closing a node, giving more resources, or more jobs to run on a node. When data comes from the app, we have three modules. Video processing, which takes the video and works with the Postgres database,
15:00
takes images out of the videos and does some interpolation and so on. We have all the images based on where a person has driven. Then we have object detection modules. There are multiple modules. We use YOLO7, and most of the data is something we have annotated ourselves, many thousands, tens of thousands of images.
15:24
Then we have different modules, some for potholes, some for cracks, and some for manholes, and so on. In this image, we also have a module that recognizes traffic science detection. We are working now on detecting damages on traffic science in general. It's quite a hard problem.
15:43
Future development. The product is under very active development at the moment. We are planning to add another class that's called registration. At the moment, we are finding damages, what we call damages. Imagine that you have a city, and then you want to map all the manholes.
16:04
Could you just drive around, find all the manholes in the images, and put them directly in the database? That's registration. All the light bulbs, for example, drive around. They take them in the image, pinpoint them on the ground, and then you have all the data instead going with the GPS and so on.
16:22
New classes that we are working on. Damages on traffic science, damages on actual manholes, and damages on street markings. A bit hard problems, but we are trying to gather a lot of data now and do annotations and so on. Clearance for vehicles.
16:40
When a tree in the spring or so, it's a bit down and cars cannot pass, we are trying to detect that. Yes. If you have any questions, you are more than welcome to write me, or we can have a coffee afterwards.