OpenDroneMap
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 188 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/31744 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer | ||
| Production Year | 2014 | |
| Production Place | Portland, Oregon, United States of America |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G 2014 Portland25 / 188
7
10
14
15
16
23
24
25
28
29
33
37
39
40
43
45
46
48
50
56
64
65
69
72
74
82
89
91
98
102
107
111
114
118
128
131
132
135
138
141
143
147
149
150
157
158
161
164
165
166
173
174
175
179
185
00:00
InternetworkingQuicksortComputer-assisted translationDigital photographyObject (grammar)Multiplication signOpen setConnected spaceModel theoryComputer animation
00:54
ForestField (computer science)Medical imagingLink (knot theory)Bit rateMetropolitan area networkKey (cryptography)Forcing (mathematics)Computer animation
01:38
BefehlsprozessorProcess (computing)Texture mappingRaster graphicsDigital photographyUniform resource locatorCrash (computing)CodeNetwork topologyPoint (geometry)Real numberDrum memoryPresentation of a groupQuantum32-bitArithmetic progressionComputer fileData structureChainClassical physicsDocument management systemLevel (video gaming)Integrated development environmentOrthogonalityDigitizingImmersion (album)Projective planeQuicksortOpen setComputer wormDisk read-and-write headPixel1 (number)Partial derivativeMatching (graph theory)NeuroinformatikSequenceMachine visionGeometryWrapper (data mining)SurfaceMedical imagingWeb 2.0Polygon meshGoodness of fitRight angleTelecommunicationFilter <Stochastik>Order (biology)Cellular automatonFrame problemQuantum statePosition operatorApproximationFigurate numberBuildingPoint cloudServer (computing)Model theoryEndliche ModelltheorieYouTubeSpacetimeGroup actionStress (mechanics)Multiplication signScripting languageVirtual machineSound effectBitFunction (mathematics)MathematicsView (database)Metropolitan area networkAssociative propertyPlotterDirectory serviceGraph coloringCausalityRadical (chemistry)Sparse matrixMenu (computing)Term (mathematics)Single-precision floating-point formatAuthorizationInferenceProduct (business)Video gameObservational studyPopulation densityService (economics)Computer configurationAreaComputer animation
11:25
BitModel theoryCrash (computing)Coordinate systemLine (geometry)Image resolutionModulare ProgrammierungTexture mappingGeometrySimilarity (geometry)Digital photographyAdditionLevel (video gaming)Multiplication signSubsetCartesian coordinate systemBasis <Mathematik>Linear equationNatural numberRepository (publishing)Set (mathematics)Google Street ViewOrder (biology)Web browserFood energySurfaceCompilation albumPoint (geometry)Virtual machineData compressionLibrary (computing)Process (computing)Data structureProjective planeDistribution (mathematics)Open setType theoryPolygon meshProduct (business)OrthogonalityExtension (kinesiology)IdentifiabilityMetropolitan area networkGraph coloringSystem identificationDemo (music)Patch (Unix)MathematicsDegree (graph theory)Real numberGoodness of fitPosition operatorVertex (graph theory)QuicksortSpacetimePresentation of a groupComplex (psychology)Spring (hydrology)EDIFMassAreaNumberPoint cloudIncidence algebraGodServer (computing)Physical systemObject (grammar)Scripting languageSocial classDuality (mathematics)Data storage deviceWeb 2.0Matching (graph theory)Endliche ModelltheorieJava appletPressureConnected spaceSatelliteSingle-precision floating-point formatRoutingWebsiteSeries (mathematics)Computer animation
21:11
Level (video gaming)Open setNetwork topologyMusical ensembleSound effectPattern languagePoint (geometry)Uniform resource locatorComputer animation
22:02
ApproximationModel theoryScripting languageLinearizationProcess (computing)Polygon meshUser interfaceProxy serverPoint cloudParameter (computer programming)RotationExecution unitTransformation (genetics)Flow separationImage resolutionComputer programmingMedical imagingMatching (graph theory)MereologyInformationGame controllerCondition numberOpen setUnitärer OperatorSpline (mathematics)Library (computing)DatabaseDemosceneVideoconferencingQuicksortAuthorizationSound effect1 (number)Wave packetGoodness of fitTerm (mathematics)Core dumpVideo gameWebsiteDivision (mathematics)Point (geometry)Computer wormLevel (video gaming)Software development kitHacker (term)Pairwise comparisonLecture/Conference
Transcript: English(auto-generated)
00:00
One of his problems, I can't remember what it was. It was a little ADD myself. Oh, yeah, yeah, that's right. He chases shiny objects. So my problem is actually similar. I sometimes chase objects that don't yet exist for reasons I don't know, sort of solutions
00:23
looking for a problem. And to prep for this, I just searched for a bunch of cat photos because that's what the internet is for. And this sometimes makes me do silly things. I didn't do this, by the way. So bear with me. I'm going to talk about a solution
00:40
to a problem that didn't exist until recently, and that's OpenDroneMap. Going back in time, I apologize. I can't actually see my screen here. So going back in time, back in time, there we go. So back in 2004, I was working on what I call my non-PhD in forest ecology, my non-PhD,
01:00
because I did three field seasons and then bailed out. But some of the fun stuff I got to do while I was there was I got to fly a big balloon. Actually, that's almost to scale right there. I think it was an 11-foot balloon. It had a lot of lift, which now that I've read the FAA restrictions on tethered balloon.
01:21
Anyway, don't buy an 11-foot balloon, OK? Six feet, that's as big as you need. Unless you want to get the FAA involved. But that never happened. So I flew a large balloon with an array of cameras, and I took a bunch of images. The array had some modified cameras.
01:43
There was a nice Kodak DCS. Oh, what was that? It was a beautiful camera. It was a pro sports camera, so it had amazing ISO levels. And unlike now, it wasn't some filtering process which sort of smoothed and removed the pixels artificially. No, each pixel had no extra electronics
02:02
associated with it. So the whole pixel was the actual capacitance cell. So all the light hitting that, and this is a full frame, this is a full frame CCD. All the light hitting that got back. And so here it is. I think it was manufactured in 2001 or 2002.
02:21
It was able to do ISOs of 4,800 and 9,600. So really, really fast camera for the time. Also, I bought it a few years after it came out, so I didn't have to spend $14,000 on it. So then put some filters in front of it, so a near infrared filter.
02:41
We had some bandpass filters. Those bandpass filters were in order to do biological research. We wanted to understand, OK, what's happening with the leaves at the leaf level? Is there light stress? So it was lots of fun stuff. Total failure of a project. Way too early, because there was no trivial way to figure out where we were taking pictures of.
03:02
And so I started looking at computer vision. We got all excited about it, and then realized that neither had the computers nor the code to get any of it done. So a few years ago, I saw a presentation. I think it was a keynote that Peter Batty gave at some geo conference of some sort. And he was talking about the future of geo.
03:20
Or maybe it might have been. Actually, it was probably. Well, anyway, whatever it was. He was talking about the future of geo, and wearable devices, and always on cameras, and the reconstruction of 3D space, and sort of all the things that could come to be. And it started getting, it sort of was a worm in my head. It sort of reminded me of the problem
03:40
that I couldn't solve before, which was how to take these arbitrary photos that come from whatever, that may not have GPS associated with them, and construct them into something that we can use in the geospatial world. So let me switch over to the presentation here. Open drone map.
04:04
So the idea here is, well, first of all, the name. It's not really a drone. It's not really a map. And for those of you who really like BSD and Apache and all the relatively liberal licenses,
04:21
it's open like the copyleft. So open drone map. It's a well-named project. So the idea is that we go from those arbitrary images through some sort of processing pipeline into point clouds, digital surface models, textured digital surface models. So this is like a full immersive 3D environment and ortho imagery. Wow, nifty stuff.
04:40
And then, of course, you can drive three DMs and classic raster surface models. And so then the question becomes, how do we get there? And so, well, actually, the first question is, what is it? So it's a processing tool chain for civilian UAS imagery. Very simple. Based on three basic technologies.
05:01
And actually, I can't remember what the acronym stand for. So look those up. Bundler, PMVS, and CMVS. These are all in this old computer vision code. This is stuff that's well-referenced, well-solved. It also means that it outputs a bunch of text files and the data structures are a mess. So we'll have to fix that. But it's really, really cool stuff.
05:21
And it's currently deployable on Ubuntu 12.04. 32-bit. Don't put it on 64-bit. Don't put it on 14.04. But we'll fix that. Don't worry. And yeah, it's a work in progress. So what it's not, it's not complete. But it's actually, there were some quantum leaps this week. Right now, what's in place is a full 3D point cloud.
05:45
It's not. I'm sorry, I should say. What we're not trying to do is replicate things that MeshLab and Cloud Compare have done as far as full 3D point cloud and mesh editing environments where you can do fancy complicated things. There's good tools for that. And MeshLab, admittedly, is a little hard to use.
06:02
But there's a bazillion YouTube videos showing you exactly how to do it. So there's at least some pedagogy for getting you there. And it's not flight planning software. And it doesn't solve everything. Also, it's not yet complete.
06:21
We still need to build in mesh creation, textured mesh creation, and orthophotos and all the associated things. But that's really close. So in the short term, what it'll be is essentially a single command you run on the command line. And you pipe your images through. And you get a bunch of files. In the long term, you might wrap it in a web front end.
06:44
You might put an easy sort of Python wrapper in QGIS for running it. And of course, you still have terminal access. And then data structuring is something that will need to be addressed, whether you push this into PostGIS or some sort of file based data structures.
07:01
So we won't go through how it works yet. But let me take you through the sequence. And this is dark. And I apologize. But the rest are not. So first, take all your photos. And you do full stereo matching. Or you do partial stereo matching. So you're basically just trying to figure out which ones are related to each other. That gives you a sparse point cloud.
07:21
Then you do full stereo matching, where you compare every pixel to every possible pixel and every other possible image that's related. And you get a dense point cloud. And then you mesh it. So you turn it into a surface. And then you start painting that surface with your original images. And well, maybe that's not the ideal way to do it. Maybe we need to do some blending.
07:41
And we need to do some other things to make that look great. But we can do that. That's there. We have the technology. And that's pretty much it. Now you've got a textured surface model. Now we can take the next steps in creating secondary products, the DEM, digital surface model, run through some classifiers to separate out the buildings from the trees from the ground,
08:01
and create DEMs and such. So let's do a quick demonstration, which is going to be really tricky. I really do like you guys. That's not why my back's to you.
08:23
OK, so the wonders of a live demonstration. Here we go. OK, so dense point cloud. Essentially, this point cloud is constructed from these original images. And if we wanted to see what those original images looked
08:42
like, there's the original image. There's the dense point cloud. This you can get now from OpenDrone Map. So feed a bunch of overlapping images in, and you will get a nice dense point cloud. And much like Aaron demonstrated yesterday or talked about yesterday, if you
09:01
want to keep your coffee warm while you're running it, you will have that option. But you can also run it on Amazon and keep their server rooms warm. So those are the actual original photos. And we can see, let's look at a couple of them that are adjacent. And again, we're doing feature matching between them.
09:21
So this tree here is the same as this tree here. Extract some matching points. Figure out how they're related. And then we do fancy magic where you're doing some linear approximation of where the rays intersect. So you can reconstruct the original camera positions and the original points. Don't make me go into it because I don't actually
09:41
fully understand it. I can just say that much, one sentence. So OK, we've got a point cloud. That's pretty nifty. Now what would be really nice, what would be really nice is a three-dimensional mesh. And this, we need to do some improvement in meshing algorithms. So we're doing the trees separately from the buildings.
10:00
But there's our buildings there and the adjacent trees. We've got some artifacts, some holes that don't exist. Oh, and I should say, this is Seneca County, Ohio. I grew up in Northwest Ohio. It's hard to show just how flat it is, but that's real.
10:22
And those ditches are about somewhere between 8 and 12 feet deep. So that gives you a sense of how much topography there is in the rest of the landscape as well. So maybe next time, we'll do more, a less flat location. So this minus meshing can be done now. The rest can actually be done in MeshLab for the time being.
10:43
And we'll work on getting that scripted into the rest and the code pulled in. But essentially, we can take those photos and we just run this little texturing from registered rasters. And this does our drape back. So now I'm going to do a high CPU process live
11:01
in front of a studio audience. We'll see what happens. I guarantee a computer crash. So it'll be hand-waving for the rest of the presentation. I do have a song prepared. Does anyone here want to hear a song? What about dance? What's that? What about dance? Dance?
11:23
Only if someone's willing to play the drums or the guitar. OK, so it worked. Yay. I don't have to sing. You guys don't have to bear with me. So there it is textured. This is actually in order to, I cheated. This is not a full texturing. This is a low resolution texturing. If we did a full resolution texturing,
11:41
it'd be even prettier. You'd be able to see a center line on that road quite nicely in full resolution. But pretty snazzy. We've now textured it back to the model. Now the next steps will be how do we then orthorectify it? How do we then georeference it? But those are comparatively easy problems. And actually, I'm going to run that.
12:01
Just because that didn't crash, we're going to run it one more time. Let's see if we can crash this. I really like singing.
12:21
I like dancing, too. Do we have any drummers in the room? Anyone with a guitar? Hopefully it doesn't crash. This is all GPU intensive and I've got an Ultrabook. All right, so now we have a little bit more detail on our model. We can zoom in and see that line a little bit better.
12:41
If we did a full texturing, this particular model would have an on the ground resolution of about two inches. So there's some interesting things you can do with that. So I work for Metropolitan Park District. And one of the things that we're going to do with this, there's two basic projects we're going to do with this. One is we're going to fly a number of natural areas,
13:03
including a wetland at the south end of our district, and identify invasive plant extents for our strike team and monitor that over time. We've never really had a good way to really monitor the extents. We also haven't had a good way to identify where those tiny little patches are
13:21
that didn't quite die off. You send a strike team out. They spray a whole bunch of poison. Stuff dies. You feel pretty good. You move on to the next place where you're going to spray poison. And one little plant just survives. And with the invasive plants, they take over and it's a mess. And of course, you just killed everything. So it's a perfect opportunity
13:41
for that invasive plant to come back. So that's one application we'll do with a wetland. And then the other thing we're going to do, because we can get one inch imagery and color infrared, we will be doing identification of native communities. So in this large wetland complex, it's very difficult to access, very difficult to walk through, very difficult to map. We'll have a sense of where some of those
14:00
really interesting communities are and potentially find some of the rare things that we know we should be good stewards of and be paying attention to and be spraying the heck out of all the invasive plants that are anywhere nearby. So that's the natural resource application. The other application, which is not as interesting to me
14:23
from the kind of project it is, but more interesting to me from the technical standpoint will be inventory of engineering structures that move over time, engineering structures along the lake. And so we'll be able to hopefully get some really good vertical accuracy
14:41
and watch for changes. We'll send our surveyors out to take individual points, we'll lock these things down in real 3D space, and then we can measure movement, not just movement of points that are registered that we send our surveyors out to periodically, that's useful, we need them, but then also find out what's happening in between.
15:04
Okay, so the demo succeeded, so no song yet. Let's see demonstration, GPS photos. Okay, so there's another application here. So this is drones, open drone map. Again, I said it's not open as in BSD, it's not drone as in killing people,
15:22
it's not really a map. So it's actually better named than open street map because it's even less what it sounds like it is. And that's what I was aiming for. So let's go a little deeper here and do something even less like open drone map.
15:44
Well, first of all, so question distribution. So how do we then distribute this? How do we share this? And there's some open-ended questions here, but one of the things that we wanna do is push the ortho products from this into open aerial map and then there are a couple of another,
16:01
there's the genesis of a couple of other DEM, Digital Surface Model mesh type projects trying to collect global data sets of these that are just sort of in the genesis stages and those will hopefully be the repositories for all the data that people process with this should they choose to. But what's really cool is folks like Howard Butler
16:23
and oh, shoot, well, okay, I'm not gonna get all the names, but Howard and folks related to him are doing things like taking C++ last zip compression libraries, so LIDAR compression libraries and porting them into JavaScript
16:41
by using JavaScript as the compilation target. Crazy stuff, shouldn't be done, has been done, is wonderful. So now you can actually throw a zipped, a last zipped, I should say, a point cloud,
17:03
basically zipped as a last object and throw it on an Amazon machine somewhere and actually view it in your browser, have it stream over as a few megabytes, few hundred megabytes instead of the few gigabytes that it really is. And then inside the browser it decompresses and you can display it with WebGL.
17:22
So not only can you easily store this and give it to people, but they can easily view it and use it. Okay, a quick plug and aside. I went to Phosphor-G Korea a couple weeks ago.
17:40
Folks there are wonderful next year. It's in Korea, you should go and you should be at least as silly as I was. It's a very, very, very beautiful place with beautiful people, a lot of energy on the geo side and a lot of fun.
18:01
The reason I bring that up is while I was there I was working with something called Mapillary. Oh, I don't have a Mapillary thing here. Hold on a second, Mapillary. So Mapillary is, for those of you who haven't seen it, it's intended to be a global crowd-sourced Google Street View product, pretty liberally licensed.
18:24
It's similar license, well compatible license with OpenStreetMap and they do these wonderful things where, well I don't have, oh yeah, there we go. So there's my little Street View from Gogum, I can't pronounce any Korean, every time I tried there was laughter.
18:41
Very nice laughter, but nonetheless. Anyway, from the tech district in Korea. So I've got this series, but you'll notice, actually if you look at this line, I did not have soju before I walked this. I want to point this out, I was actually perfectly sober, at least as sober as I am on an ordinary basis
19:01
and I certainly had not consumed any alcohol within I think probably a week of when I did this walk. And you can see it just going all over the place because I had this GPS enabled camera that was just not that good. So the photos are great, but the GPS wasn't that great. So I said, well this is a problem
19:21
for OpenDroneMap naturally. Now I didn't have OpenDroneMap far enough along at that time to actually use it, so I might have used a commercial software package to do this, but we won't talk about that. So here's a subset of those points. They look perfectly fine, I mean other than not being in a straight line because I was actually walking along a sidewalk along a road. So if we take them into QGIS and just do a quick heat map we can see we've got duplicate points, right?
19:43
This should be just all one continuous red, but we can see this is like one point, this is like two points, this is like, I don't know how many points, and then you get here and you got, basically every time I took a photo it wasn't properly updating the ephemeris from the GPS and so in addition to being wrong, it was really wrong. So feed that through that structure from motion approach
20:03
so we take each of those photos, we match them to each other, we match similar features in each, and now we know the relative camera positions all the way along that sidewalk. So now we've got, not in real space, not in any coordinate system that we know, we've got, but we've got some sense
20:21
of what the spacing of those camera positions were. So cool, that's fun, what can we do with that? Well, there we go, there it is, there it is sort of geographically. It runs at 45 degrees because we're probably, the model's somehow optimizing it to minimize the X and the Y axis.
20:42
So what we do is we actually figure out what the local coordinates are at the end points. So look at this end point and this end point and do a quick linear equation to express that and now we've got it actually geo-referenced, nifty. Now we run it through a heat map and we see everything looks about the same, it's even, we've now gone from this set of points to,
21:03
from this set of points to nicely spread out points and we shifted our geography as well. And so if we, never, I'm a terrible cartographer, let's be honest. So here it is on top of the map. Here's the original points, nowhere near Tehran Road
21:23
and here they are in the correct location, perfectly spaced. And even with those little places where I had to walk around groups of people or trees as I was switching from across the road. So open drone map, not just drones, not just maps, in fact, not drones, not maps and open as in GPL.
21:44
Thank you.
22:01
I'm actually thinking from the ground-based standpoint, could you take a movie camera and just sort of go around and then dump out individual JPEGs with VLC and then take those overlapping images and do the 3D effect? Yes, you can. You want a good lens, so avoid your cell phone.
22:23
But otherwise, yeah. And low resolution isn't necessarily a bad thing. So although like with a 4K camera, you're not really talking about very low resolution images. Well, I'm thinking HD video. Yeah, yeah, HD should work. Yeah, as long as you have a decent lens.
22:42
And the other caveat is there may be some work. So for most sort of consumer cameras, there's an existing database of what the size of the CMOS chip is and what the focal distance is and the relevant photographic parameters.
23:02
So there may be a calibration process or some things you'll need to look up. Usually when I look up stuff and I don't know, I haven't done it with video cameras, but I think DigiCam has a library built into it for a lot of that. Oh, does it? Yeah. Okay, with that information. What's that? Lens fun. Lens fun? Cool, so then you could actually add those.
23:31
Oh, well. Anyway, search, search.
23:44
Wow, train of thought derailed. Yeah, so you can actually, there's, and when I write the documentation in the next week or two, there's a place where then you can add those parameters to the calibration information in the open drone map
24:00
and then run it, so. Thanks for the talk. I'm curious about control. Is it internal or is it not part of the program? The program, so you mentioned sending your surveyors out.
24:21
When they come back with some control points, are you doing the matching in a separate thing from open drone or is the control built into? Well, it hasn't been built yet, so. As a starting place, we'll probably pass it as parameters on the command line. It'll probably be a two-step process then.
24:40
Run through, create the point cloud, the mesh, et cetera, and then you'd run a separate script to do, basically pass in. We'll start with just a linear approximation model, 3D transformation, probably just, yeah, just affine transformation, so rotation, scaling, skewing, and then we can iterate from there.
25:01
From the user interface side, obviously that's a user interface, but that's not everyone's favorite thing. I don't know exactly how that'll work. Part of the reason for the, what are the core competencies of this is, I hope it'll be a core pipeline and then be something people can build tools around. We will build some tools around it as well,
25:21
obviously, for our use. And also, in the short term, you can easily georeference point cloud in Cloud Compare. Cloud Compare is awesome. And it's another one of those, not particularly intuitive, but great tutorials online, packages. And it's not fair to say that about MeshLab or Cloud Compare. They do freaking everything in the world.
25:41
There's maybe no good user experience that can be created from something that does everything. But question here. What are you doing for your do-it-yourself infrared channel?
26:02
I haven't done anything yet. Wasn't that a infrared? Yeah, that camera's hard to find now. Yeah, that camera was made in 2001. You can still get it. I remember it. It's a DSC720X, and the X, because it was really, really fast.
26:21
And you can still get it on eBay. And if you're lucky, you can get it for under 1,000 actuations. There's a few of them hanging out there. I have a persistent search there. I have not asked my wife if I can buy one. Yeah, that's why the search is still there. One day, one day, maybe there'll be a little extra money and I can buy one.
26:41
The public labs guys, I don't know if you've seen their stuff. I was just curious if you were doing their film hack. I'm totally gonna buy one of their kits. They just revised their kits in the last couple weeks. And yeah, yeah, we're gonna buy their kits. We're gonna throw them up on balloons. We now have authorization to fly from the FAA
27:01
for a couple of sites, but there's gonna be a lot of stuff we wanna fly that the authorization will take too long. And we just, it's an opportunity and we wanna get out there. So we'll throw up a six foot balloon, not an 11 foot balloon, a six foot balloon, and we'll put a payload on it and we'll, yeah. Actually with their cameras, we can probably put up like a three or four foot balloon.
27:20
So thanks. Yeah, public labs is awesome. That's a great way to get a bird in the air. Cool, thanks so much. Thank you.