An open API for 3D-georeferenced historical pictures
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 351 | |
Author | ||
License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/68994 (DOI) | |
Publisher | ||
Release Date | ||
Language | ||
Production Year | 2022 |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
| |
Keywords |
FOSS4G Firenze 2022249 / 351
1
7
13
22
25
31
33
36
39
41
43
44
46
52
53
55
58
59
60
76
80
93
98
104
108
127
128
133
135
141
142
143
150
151
168
173
176
178
190
196
200
201
202
204
211
219
225
226
236
242
251
258
263
270
284
285
292
00:00
Open setHypermediaProjective planePresentation of a groupUniverse (mathematics)Computer animation
00:19
Vector potentialInstance (computer science)SatelliteSource codeMultiplication signRight angleDigital photography
00:56
Sound effectNatural numberInstance (computer science)Right angleHazard (2005 film)Mathematical analysis
01:22
Order (biology)NeuroinformatikFile archiverGeometryComputing platformAlgorithmComputer animation
02:05
Computer-generated imageryMoment (mathematics)MereologyComputing platformPhysical systemPresentation of a groupMedical imagingComputer animation
03:05
DigitizingVirtual realityControl flowComputer-generated imageryPoint (geometry)Computing platformAreaFile viewerCone penetration testAzimuthOpen setDependent and independent variablesContent (media)Personal digital assistantRevision controlField (computer science)File formatParameter (computer programming)Direction (geometry)Atomic numberStandard deviationSurfacePosition operatorSoftware frameworkImplementationState transition systemGraph (mathematics)Linked dataMobile WebHypothesisAlgorithmCalculationDistortion (mathematics)BildtelegraphiePerspective (visual)Machine learningOrdinary differential equationPoint cloudImplementationAngleDatabasePosition operatorProjective planeOrder (biology)Subject indexingMedical imagingDigital photographyPoint (geometry)View (database)MetadataServer (computing)Standard deviationCASE <Informatik>Software frameworkMeasurementDigitizingGraph (mathematics)Group actionGame controllerNetwork topologyQuantum stateMobile appUniform resource locatorObservational studyPixelGreatest elementSubsetAlgorithmRoutingDistortion (mathematics)Mathematical analysisDot productSpline (mathematics)SoftwareBinary fileProcess (computing)Motion captureCone penetration testReliefRange (statistics)Self-organizationBitMereologySoftware testingSet (mathematics)Computer fileScripting languageLatent heatMachine learningCore dumpINTEGRALSpacetimeField (computer science)Open setAttribute grammarUniverse (mathematics)Associative propertyWeb 2.0Revision controlCalculationOrientation (vector space)Student's t-testMathematicsResultantWave packetAzimuthVirtual machineLink (knot theory)Right angleDifferent (Kate Ryan album)State of matterPhysical systemCartesian coordinate systemWordLinked dataSmartphoneOpen sourcePrototypeInstance (computer science)Slide rulePerspective (visual)GeometryFlow separationCloud computingCompass (drafting)Electronic visual displayAreaFrequencyState observerRootRouter (computing)Row (database)Point cloudLevel (video gaming)MetreComputing platformComputer animation
Transcript: English(auto-generated)
00:00
My name is Jens. We're from the University of Applied Sciences, Western Switzerland, and we have done this project together also with the University of Zurich. So during the presentation we will switch. I will start and then hand over to Nicola. So this morning and right before we saw some things that we can do with 2D geo-referenced
00:28
pictures, satellite imagery for instance. But there is also a huge potential in 3D geo-referenced pictures, because 3D geo-referenced pictures, especially photos,
00:42
they go back in time much further than satellite imagery. So we can also take this as a data source to do lots of things. So for instance to analyze the retreat of glaciers, or we can also see the effects of urbanization. For instance on the left we see historical pictures, a picture from the
01:06
1950s, and on the right we see a more recent one. Or also natural hazards, that's another thing where we can use 3D geo-referenced imagery to do some analysis.
01:21
However, 3D geo-referenced historical pictures, they are often distributed in many archives around the world. One problem is that these pictures are generally not digitized, so in order to use them with the computer you need to put them on the scanner. Then of course
01:42
pictures are not geo-referenced, in order to use them with different algorithms and so on, it's good to have geo-reference. And often pictures are not properly tagged, so there are pictures but nobody knows exactly where they have been taken, when they have been taken, and
02:00
what is visible on these pictures. So some years ago we have created a platform called Snapshot, and on this platform we use crowdsourcing for the 3D geo-referencing of images. So you can see, if you want to check yourself, here's the URL of the system, and there's a
02:23
screenshot of the system where you can see a 3D geo-reference picture, a historical picture superimposed on a virtual globe. So at the moment we have around more than 700 volunteers, and we have more than 200,000 geo-referenced images coming from collections from
02:44
all over Switzerland, part of Austria and also Brazil. I will now hand over the presentation to Nicolas who will explain to you how Snapshot works and also what is the goal of the open API that we have developed. Thank you Jens. So when we are speaking about geo-referencing images,
03:06
actually you may not be familiar with that wording, but when you are taking a picture with your camera or your smartphone, it has an actual position on the earth. So the picture that has been taken by that camera could be virtually geo-referenced by selecting points, which are
03:24
called ground control points within the photogrammetry field, on the picture itself, and on a 3D globe. We are based on a 3D globe for the geo-referencing process in Snapshot, and this allows you to compute the pose, well not actually of the image itself, but of the camera that has taken that
03:43
image. So it will give you the three values for the position on the earth and then the three angles around the same axis, the azimuth, the pitch, and the world angle. And for those who are not familiar with projecting themselves into these 3D views, we can understand the same principle
04:01
by these small 2D sketches. Like for example on the first one, you see the two dots that were clicked on the image, which are the ground control points, and on the terrain actually, on the trees and the house. And by rectifying the 3D rays that originate from the camera
04:22
position to the clicked point on the 2D space of the image and on the ground, you can align the image correctly in the 3D world. And based on that, we can further explore the data once the picture has been geolocalized. And for example, we are able to compute its footprint, so
04:41
the projection of the border of the image on the ground, which is then useful for computing other metadata and extracting, for example, place names, which you can then select as an index tool for searching, for example, images from certain mountains or certain rivers that you are interested in for such studies, for example. So on the really first
05:05
raw metadata such as these, we have what we call a view cone, which is actually not really a cone there because it's projected on the map. And it's really the field of view of the picture from the position, and that's it. You may define the maximum range if you want, but it does not
05:20
necessarily have to have one. And after that, we compute the footprint. Now it's directly done into cesium, so it's basically the projection of the border of the image on the ground. And after that, we are using a tool called the viewshed analysis from a software called saga GIS, which is open source as well, to compute the viewshed. So it's basically
05:42
taking into account the relief in the image to show you what are exactly the pixels that are visible on the ground. So if you have a valley and there are houses or river on the bottom of that valley, it will be hidden by the foreground mountain, so it won't show up. So with all these, we have done after a study with the Swiss Confederation work to improve our
06:07
first API, which was a bit messy, to reorganize it and to provide it as a... Actually, you can follow this URL. It will show you a swagger also known as the open API version 3. So it's
06:21
really well documented. You have many endpoints to select images based on certain attributes and so on. So this really opened doors to many use cases within universities or whatever organization which want to do some analysis of these images, of these historical images within Switzerland, a bit of Austria, as Jens said before, and Brazil today. And it also gives the users a
06:44
standard access to the database, because prior to that, we have plenty of partners who have tests to feed them some data sets or extract of data sets, so we have to do ourselves some dump from the database, then they organize them into CSV file and things like that. But now it's all done through the API, actually, which is really cool.
07:03
And we are exploring now more way of exporting this data in standard way. For example, we have seen that within the EXIF data that you may have on your photos, the position is given, for example, if you have a GPS chip connected to your camera, but the angles are not supported,
07:23
except for the azimuth using the compass, but it's not really precise. And we actually use DLTF for displaying onto the Cesium globe on the web platform, which is encoded into JSON or GLB for the binary format. And it can contain the pose of the image, but it's a bit messy to understand the projection system, well, not really the projection system, but the order
07:45
of the orientation angles within the Cesium framework. And it's only supported in certain amount of tool, not necessarily all. And we are also digging into the newly IIIF server for
08:00
serving large images. For example, user are using IIIF to serve digital pieces of arts images, for example, and things like that. And you can really zoom into the images, it is served as tiled images, which would be optimized for the web, but it has not yet support for geodata. They are actually working actively on trying to integrate and figure out
08:22
how to get geolocated within the IIIF, but it's not yet finalized. And we also are much into the work of the OGC geopause standard working group to try to figure out how we can integrate that to serve, for example, a geopause, which is compliant with the OGC standard within
08:42
our API. And this will allow us normally to be interoperable with any other software that will implement the geopause standard. So this is just on the right, a small capture of the Swagger API that you may find on the URL I shared before. And underneath, there is a small subset of the
09:02
camera position and its footprint, where you can see, for example, in one of the routes, I don't remember exactly which one, but if you want, we can discuss after. The pose, which is returned by a route of the API for a given photo. So we have the altitude in meters, latitude longitude as angles, and all the three angles plus the focal in pixel of the camera.
09:24
And the background implementation is using Postgres and PostGIS for storing the data in the database. We are using Python for background script. Of course, JavaScript for the web parts and the open API specification already did talk about it, the GLTF and IIIF, because we are now also able to integrate images from people and institutions who has already
09:45
in, we are already serving images as IIIF data. So we are able to connect to the server, made the people geo reference the picture and then feed them back the data so that they can, the owner of the collection can integrate the result of the geolocation by the volunteers.
10:04
So I will give up the talk to Jens back for the last slides. Thank you, Nicolas. So now that we have this API, we have some use cases where different people, different organizations are already using this API to do different things.
10:27
One use case is a project of the University of Zurich. The project is called Bilder der Zweitz online, so images of Switzerland online. They have paintings, both paintings and photos,
10:41
and they use this new API for the extraction of data. And their goal is to build a knowledge graph using linked data. This means that, for instance, if they have some famous painters who have painted some pictures of Switzerland, they know when it has been painted. But for instance, they know, they don't know exactly from where the picture has been painted.
11:03
So snapshot is used to generate this metadata to compute the exact location, and so they can retrace the way of the painters in their knowledge graph. And something that we also implemented in this API is the pushing of new images to snapshot through this API.
11:26
So once they have a new collection or they have new pictures in a collection, they can simply push the IIIF links through the API to snapshot. The pictures are geo-referenced by the volunteers, and afterwards they can re-access the new data that has been generated.
11:49
Another use case is re-photography. So this is the idea of retaking a picture, a historic picture, but today. So there were two students, two master students from the
12:04
University of Lund who implemented a prototype, which you can see here on the left. It's a mobile application that shows you a historical picture that shows you how to get to the point from where the historical picture has been taken. Of course, these are only terrestrial
12:25
photos that have been selected in order to be able to retake the photo. And once you have retaken the photo, they calculate some measures of accuracy in order to indicate to the user how accurate the re-photograph is. Another use case that has been recently implemented is
12:50
the automatic positioning of a UAV, so a drone. So there were some scripts that have been implemented in order to automatically position a UAV, a drone, at a certain point and to retake
13:04
the historical pictures. So now these two use cases, you can see that it's possible to retake both terrestrial photos and also aerial photos, oblique aerial photos. Another use case that also came up while we were discussing with the University of Zurich is,
13:29
for instance, the calculation of artistic distortion. So if you geo-reference a painting, of course, geometrically, it's not exactly the same as a photo. And in the 18th, 19th century,
13:45
there were many painters who thought that they could make the terrain look more dramatic, so they exaggerated the terrain. And this you can see on the picture on the right. Another use case is also the training of algorithms, so machine learning algorithms
14:04
for the automated detection of landscape features. There's also the EPFL who contacted us, who want to implement a machine learning algorithm that takes these pictures and is able to recognize
14:23
different features in the pictures. And since we have the geolocation, we can exactly have the geolocation of these different features. And the idea is then to position a drone without the GPS, just by recognizing the different features in the pictures that comes from the drone's camera.
14:45
And third use case, there are some projects all over the world that are called photographic observatories. Here, the idea is also to do re-photography from specific points at a certain frequency, for instance, every year or every second year.
15:06
And then to document and analyze landscape change. And now as a perspective, now that we have this open API, which is connected to
15:22
Smashot, we would like to go towards a decentralized infrastructure for 3D geo-referenced imagery. So with Smashot, it's possible to geo-reference a picture in 3D, but it's of course also possible to do geo-referencing using automatic geo-referencing. So this is also
15:42
something that Nicolas has done some years ago, where we have one picture where we know exactly the geolocation, and we are able to detect in another picture that does not have a geolocation matching points. And since we have the geolocation of these matching points, we can automatically geo-reference a second image, and so on. If we have several images that are more or
16:06
less taken on the same place, we can use the existing geo-reference of existing photos to geo-reference pictures, photos without a geo-reference. And of course, as Nicolas already said,
16:23
we also want to use geo-post for future implementations, for future versions of this API. So this new OGC standard and also OGC API features is also something very interesting. And we aim at creating a consortium of state entities, archives, universities, and associations
16:44
to create this decentralized infrastructure and to focus on standardization, but also on cloud-based infrastructures. Okay, thank you very much for listening.