We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

An open API for 3D-georeferenced historical pictures

00:00

Formal Metadata

Title
An open API for 3D-georeferenced historical pictures
Title of Series
Number of Parts
351
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language
Production Year2022

Content Metadata

Subject Area
Genre
Abstract
Approach and concepts 3D-georeferenced historical pictures have a high potential for the analysis of different landscape features such as melting glaciers, the effects of urbanization or natural hazards. Moreover, historical pictures have a higher temporal and spatial resolution than satellite imagery and therefore allow for analyses that go farther back in time. A 3D georeferenced picture can for instance be combined with a digital terrain model (DTM) and other reference data to calculate the exact footprint of the picture and to generate a list of visible toponyms that can be used to find pictures of a specific place or region. The utilization of historical pictures is unfortunately still difficult: 1. historical pictures need to be digitized 2. collections are often spread across several places in different archives and collections 3. metadata is often not available. In the ongoing open-source project Smapshot (Produit et al. (2018), smapshot.heig-vd.ch/) over 150’000 digitized historical pictures have been georeferenced in 3D by more than 700 participants. In the web-platform Smapshot a participant can georeference a picture using monoplotting (Bozzini et al, 2012): ground-control-points (GCP) are digitized both in the historical picture and in a virtual globe that displays recently updated data. These GCP allow for the calculation of the exact position from where the picture has been taken (3D point) and the three angles that define the direction of view: roll, pitch and yaw. Once the position and the direction of view has been calculated a footprint of the picture is generated using a DTM. Results In order to make the pictures and the metadata from Smapshot available to the public, an open API for 3D-georeferenced historical pictures has been created. The goal was to offer free access to all the data in the Smapshot database and to allow for different types of queries such as retrieving the footprints of the photos, fetching metadata for a picture (e.g. owner, title, date, x/y/z position and roll, pitch, yaw angles) or retrieving photos that are within a certain range from a specific point. This API was built in NodeJS (nodejs.org/) with a PostgreSQL/PostGIS (postgresql.org/, postgis.net/) database and python code for the georeferencing algorithm. The API is a REST API fully documented using the OpenAPI specification. The API project has been open-sourced and specific test-suites have been put in place to ensure quality and to allow community contribution with confidence. One challenge for the establishment of this API was standardization: Today there are several standards for the definition of metadata in pictures such as the IIIF (iiif.io/) or the Dublin-Core (dublincore.org/) standards. These standards however have limited support for geospatial data. On the other hand, spatial standards poorly support pictures that are oriented in 3D. The glTF standard (khronos.org/gltf/) is one example and there is also a recent initiative from the OGC called GeoPose (ogc.org/projects/groups/geoposeswg) which formalizes a standard to define a 6DoF pose everywhere on Earth including a position and orientation in 3D. Reasons why it should be considered 3D georeferenced images are increasingly used by several projects that document change over time; e.g. within the field of digital humanities even paintings can be considered for 3D georeferencing and differences between the real world and the painted world can give room for analysis and interpretation. Another use-case is the creation of geovisualization-applications that show the contents of historical pictures in 3D and that enable a user to compare its contents to the real world (e.g. augmented or virtual reality applications) Furthermore in the context of climate change, pictures and paintings document change and deliver evidence. Image processing techniques can be used to automatically detect features (machine learning) and if several pictures are available for one region (but taken from slightly different viewpoints) 3D features can be generated. The open API for 3D georeferenced historical pictures makes these types of analyses easier and opens up the data for a larger public. It also becomes possible to implement other solutions that utilize the data directly - e.g. for displaying historical pictures in a 3rd party web page or for implementing machine-learning processes that automatically download pictures and metadata in order to recognize features and places. The results of the project are also an important input for standardization activities that aim at establishing standards in the context of georeferenced pictures and their metadata. An important perspective of the project is the establishment of an infrastructure for 3D-georeferenced pictures that can be deployed on a national or international level and that also offers the possibility to push new data (e.g. pictures) in the database. This work is of interest for researchers who want to utilize and analyze 3D georeferenced historical imagery and for people who want to establish open API’s to give access to data that is relevant for research.
Keywords
202
Thumbnail
1:16:05
226
242
Open setHypermediaProjective planePresentation of a groupUniverse (mathematics)Computer animation
Vector potentialInstance (computer science)SatelliteSource codeMultiplication signRight angleDigital photography
Sound effectNatural numberInstance (computer science)Right angleHazard (2005 film)Mathematical analysis
Order (biology)NeuroinformatikFile archiverGeometryComputing platformAlgorithmComputer animation
Computer-generated imageryMoment (mathematics)MereologyComputing platformPhysical systemPresentation of a groupMedical imagingComputer animation
DigitizingVirtual realityControl flowComputer-generated imageryPoint (geometry)Computing platformAreaFile viewerCone penetration testAzimuthOpen setDependent and independent variablesContent (media)Personal digital assistantRevision controlField (computer science)File formatParameter (computer programming)Direction (geometry)Atomic numberStandard deviationSurfacePosition operatorSoftware frameworkImplementationState transition systemGraph (mathematics)Linked dataMobile WebHypothesisAlgorithmCalculationDistortion (mathematics)BildtelegraphiePerspective (visual)Machine learningOrdinary differential equationPoint cloudImplementationAngleDatabasePosition operatorProjective planeOrder (biology)Subject indexingMedical imagingDigital photographyPoint (geometry)View (database)MetadataServer (computing)Standard deviationCASE <Informatik>Software frameworkMeasurementDigitizingGraph (mathematics)Group actionGame controllerNetwork topologyQuantum stateMobile appUniform resource locatorObservational studyPixelGreatest elementSubsetAlgorithmRoutingDistortion (mathematics)Mathematical analysisDot productSpline (mathematics)SoftwareBinary fileProcess (computing)Motion captureCone penetration testReliefRange (statistics)Self-organizationBitMereologySoftware testingSet (mathematics)Computer fileScripting languageLatent heatMachine learningCore dumpINTEGRALSpacetimeField (computer science)Open setAttribute grammarUniverse (mathematics)Associative propertyWeb 2.0Revision controlCalculationOrientation (vector space)Student's t-testMathematicsResultantWave packetAzimuthVirtual machineLink (knot theory)Right angleDifferent (Kate Ryan album)State of matterPhysical systemCartesian coordinate systemWordLinked dataSmartphoneOpen sourcePrototypeInstance (computer science)Slide rulePerspective (visual)GeometryFlow separationCloud computingCompass (drafting)Electronic visual displayAreaFrequencyState observerRootRouter (computing)Row (database)Point cloudLevel (video gaming)MetreComputing platformComputer animation
Transcript: English(auto-generated)
My name is Jens. We're from the University of Applied Sciences, Western Switzerland, and we have done this project together also with the University of Zurich. So during the presentation we will switch. I will start and then hand over to Nicola. So this morning and right before we saw some things that we can do with 2D geo-referenced
pictures, satellite imagery for instance. But there is also a huge potential in 3D geo-referenced pictures, because 3D geo-referenced pictures, especially photos,
they go back in time much further than satellite imagery. So we can also take this as a data source to do lots of things. So for instance to analyze the retreat of glaciers, or we can also see the effects of urbanization. For instance on the left we see historical pictures, a picture from the
1950s, and on the right we see a more recent one. Or also natural hazards, that's another thing where we can use 3D geo-referenced imagery to do some analysis.
However, 3D geo-referenced historical pictures, they are often distributed in many archives around the world. One problem is that these pictures are generally not digitized, so in order to use them with the computer you need to put them on the scanner. Then of course
pictures are not geo-referenced, in order to use them with different algorithms and so on, it's good to have geo-reference. And often pictures are not properly tagged, so there are pictures but nobody knows exactly where they have been taken, when they have been taken, and
what is visible on these pictures. So some years ago we have created a platform called Snapshot, and on this platform we use crowdsourcing for the 3D geo-referencing of images. So you can see, if you want to check yourself, here's the URL of the system, and there's a
screenshot of the system where you can see a 3D geo-reference picture, a historical picture superimposed on a virtual globe. So at the moment we have around more than 700 volunteers, and we have more than 200,000 geo-referenced images coming from collections from
all over Switzerland, part of Austria and also Brazil. I will now hand over the presentation to Nicolas who will explain to you how Snapshot works and also what is the goal of the open API that we have developed. Thank you Jens. So when we are speaking about geo-referencing images,
actually you may not be familiar with that wording, but when you are taking a picture with your camera or your smartphone, it has an actual position on the earth. So the picture that has been taken by that camera could be virtually geo-referenced by selecting points, which are
called ground control points within the photogrammetry field, on the picture itself, and on a 3D globe. We are based on a 3D globe for the geo-referencing process in Snapshot, and this allows you to compute the pose, well not actually of the image itself, but of the camera that has taken that
image. So it will give you the three values for the position on the earth and then the three angles around the same axis, the azimuth, the pitch, and the world angle. And for those who are not familiar with projecting themselves into these 3D views, we can understand the same principle
by these small 2D sketches. Like for example on the first one, you see the two dots that were clicked on the image, which are the ground control points, and on the terrain actually, on the trees and the house. And by rectifying the 3D rays that originate from the camera
position to the clicked point on the 2D space of the image and on the ground, you can align the image correctly in the 3D world. And based on that, we can further explore the data once the picture has been geolocalized. And for example, we are able to compute its footprint, so
the projection of the border of the image on the ground, which is then useful for computing other metadata and extracting, for example, place names, which you can then select as an index tool for searching, for example, images from certain mountains or certain rivers that you are interested in for such studies, for example. So on the really first
raw metadata such as these, we have what we call a view cone, which is actually not really a cone there because it's projected on the map. And it's really the field of view of the picture from the position, and that's it. You may define the maximum range if you want, but it does not
necessarily have to have one. And after that, we compute the footprint. Now it's directly done into cesium, so it's basically the projection of the border of the image on the ground. And after that, we are using a tool called the viewshed analysis from a software called saga GIS, which is open source as well, to compute the viewshed. So it's basically
taking into account the relief in the image to show you what are exactly the pixels that are visible on the ground. So if you have a valley and there are houses or river on the bottom of that valley, it will be hidden by the foreground mountain, so it won't show up. So with all these, we have done after a study with the Swiss Confederation work to improve our
first API, which was a bit messy, to reorganize it and to provide it as a... Actually, you can follow this URL. It will show you a swagger also known as the open API version 3. So it's
really well documented. You have many endpoints to select images based on certain attributes and so on. So this really opened doors to many use cases within universities or whatever organization which want to do some analysis of these images, of these historical images within Switzerland, a bit of Austria, as Jens said before, and Brazil today. And it also gives the users a
standard access to the database, because prior to that, we have plenty of partners who have tests to feed them some data sets or extract of data sets, so we have to do ourselves some dump from the database, then they organize them into CSV file and things like that. But now it's all done through the API, actually, which is really cool.
And we are exploring now more way of exporting this data in standard way. For example, we have seen that within the EXIF data that you may have on your photos, the position is given, for example, if you have a GPS chip connected to your camera, but the angles are not supported,
except for the azimuth using the compass, but it's not really precise. And we actually use DLTF for displaying onto the Cesium globe on the web platform, which is encoded into JSON or GLB for the binary format. And it can contain the pose of the image, but it's a bit messy to understand the projection system, well, not really the projection system, but the order
of the orientation angles within the Cesium framework. And it's only supported in certain amount of tool, not necessarily all. And we are also digging into the newly IIIF server for
serving large images. For example, user are using IIIF to serve digital pieces of arts images, for example, and things like that. And you can really zoom into the images, it is served as tiled images, which would be optimized for the web, but it has not yet support for geodata. They are actually working actively on trying to integrate and figure out
how to get geolocated within the IIIF, but it's not yet finalized. And we also are much into the work of the OGC geopause standard working group to try to figure out how we can integrate that to serve, for example, a geopause, which is compliant with the OGC standard within
our API. And this will allow us normally to be interoperable with any other software that will implement the geopause standard. So this is just on the right, a small capture of the Swagger API that you may find on the URL I shared before. And underneath, there is a small subset of the
camera position and its footprint, where you can see, for example, in one of the routes, I don't remember exactly which one, but if you want, we can discuss after. The pose, which is returned by a route of the API for a given photo. So we have the altitude in meters, latitude longitude as angles, and all the three angles plus the focal in pixel of the camera.
And the background implementation is using Postgres and PostGIS for storing the data in the database. We are using Python for background script. Of course, JavaScript for the web parts and the open API specification already did talk about it, the GLTF and IIIF, because we are now also able to integrate images from people and institutions who has already
in, we are already serving images as IIIF data. So we are able to connect to the server, made the people geo reference the picture and then feed them back the data so that they can, the owner of the collection can integrate the result of the geolocation by the volunteers.
So I will give up the talk to Jens back for the last slides. Thank you, Nicolas. So now that we have this API, we have some use cases where different people, different organizations are already using this API to do different things.
One use case is a project of the University of Zurich. The project is called Bilder der Zweitz online, so images of Switzerland online. They have paintings, both paintings and photos,
and they use this new API for the extraction of data. And their goal is to build a knowledge graph using linked data. This means that, for instance, if they have some famous painters who have painted some pictures of Switzerland, they know when it has been painted. But for instance, they know, they don't know exactly from where the picture has been painted.
So snapshot is used to generate this metadata to compute the exact location, and so they can retrace the way of the painters in their knowledge graph. And something that we also implemented in this API is the pushing of new images to snapshot through this API.
So once they have a new collection or they have new pictures in a collection, they can simply push the IIIF links through the API to snapshot. The pictures are geo-referenced by the volunteers, and afterwards they can re-access the new data that has been generated.
Another use case is re-photography. So this is the idea of retaking a picture, a historic picture, but today. So there were two students, two master students from the
University of Lund who implemented a prototype, which you can see here on the left. It's a mobile application that shows you a historical picture that shows you how to get to the point from where the historical picture has been taken. Of course, these are only terrestrial
photos that have been selected in order to be able to retake the photo. And once you have retaken the photo, they calculate some measures of accuracy in order to indicate to the user how accurate the re-photograph is. Another use case that has been recently implemented is
the automatic positioning of a UAV, so a drone. So there were some scripts that have been implemented in order to automatically position a UAV, a drone, at a certain point and to retake
the historical pictures. So now these two use cases, you can see that it's possible to retake both terrestrial photos and also aerial photos, oblique aerial photos. Another use case that also came up while we were discussing with the University of Zurich is,
for instance, the calculation of artistic distortion. So if you geo-reference a painting, of course, geometrically, it's not exactly the same as a photo. And in the 18th, 19th century,
there were many painters who thought that they could make the terrain look more dramatic, so they exaggerated the terrain. And this you can see on the picture on the right. Another use case is also the training of algorithms, so machine learning algorithms
for the automated detection of landscape features. There's also the EPFL who contacted us, who want to implement a machine learning algorithm that takes these pictures and is able to recognize
different features in the pictures. And since we have the geolocation, we can exactly have the geolocation of these different features. And the idea is then to position a drone without the GPS, just by recognizing the different features in the pictures that comes from the drone's camera.
And third use case, there are some projects all over the world that are called photographic observatories. Here, the idea is also to do re-photography from specific points at a certain frequency, for instance, every year or every second year.
And then to document and analyze landscape change. And now as a perspective, now that we have this open API, which is connected to
Smashot, we would like to go towards a decentralized infrastructure for 3D geo-referenced imagery. So with Smashot, it's possible to geo-reference a picture in 3D, but it's of course also possible to do geo-referencing using automatic geo-referencing. So this is also
something that Nicolas has done some years ago, where we have one picture where we know exactly the geolocation, and we are able to detect in another picture that does not have a geolocation matching points. And since we have the geolocation of these matching points, we can automatically geo-reference a second image, and so on. If we have several images that are more or
less taken on the same place, we can use the existing geo-reference of existing photos to geo-reference pictures, photos without a geo-reference. And of course, as Nicolas already said,
we also want to use geo-post for future implementations, for future versions of this API. So this new OGC standard and also OGC API features is also something very interesting. And we aim at creating a consortium of state entities, archives, universities, and associations
to create this decentralized infrastructure and to focus on standardization, but also on cloud-based infrastructures. Okay, thank you very much for listening.