Hello, my name is Sebastian Meyer, I work for the Fraunhofer Institute of Abtonics, System Technologies and Image Exploitation in Karlsruhe, Germany. The topic of my talk is outdoor AR application for the digital map table. Here is a short outline of the slides.

First I want to present the digital map table, its purpose and functions. After that I have some slides about the basics of augmented reality, GPS and some example applications that already combine those two technologies. Next I will talk how we created an application and how markers can be placed

in augmented reality and the resulting challenges. After that I will present you our evaluation and results followed by a conclusion. The digital map table is, other than the title may suggest, not only a table

but a whole software family. The abbreviation DGLT is short for the German digitalar lagetisch. The digital map table is a software system for shared situation visualization and analysis. Any number of users can work independently of each other on the same situation using personal computers and tablets alongside shared digital tables,

that's where the name came from, or large screens. The latest addition is DGLT VR, a virtual environment where people at different locations can work together in a virtual environment. The underlying software is modular and can easily be custom tailored towards specific needs and extended depending on the requirements.

Its uses range from educational use to mission preparation, mission execution and review. A diverse range of data sources and geodata can be integrated to provide the right information for each use case. This provides a basis to correctly judge the situation and make the right decisions.

To further support users on their mobile devices in the field, we wanted to show information not only on the map, but utilize augmented reality to overlay information directly in the view of the user. We all knew that the technology exists and how accurate it should be when we all use navigation systems in our cars,

but how good it would work for our use case we had to find out. First, some words about augmented reality or short AR. AR is defined as the combination of the real world with the virtual world. The interaction has to be in real time and the environment has to be registered in 3D.

On the lower right, you see how that works. Small reference points are detected and tracked in the camera video. This can also be a depth image, for example, in the later iPhones. An algorithm then tries to detect planes in the tracked points and tries to detect walls and floors.

This works best if a lot of points are detected. The shown brick wall works perfect, a white shadowless office wall not so much. These walls are placed in a coordinate system and then you can render arbitrary objects in the scene like the little rocket on the table.

Luckily for us, the algorithms are already implemented and work very well. There is ARKit from Apple for all Apple devices like iPhone and iPad, ARCore from Google for all Android devices and even third-party implementations like Vuforia.

The next technology we use is the global navigation satellite system, short GNSS. GNSS consists of the systems Navstar from the US, GLONASS, the Russian system, Galileo, the European system, and Baidu, the Chinese system. I don't know about other countries, but in Germany, no one uses the term GNSS.

We all just say GPS. With all systems, there are over 100 satellites in orbit and you need to receive signals from four to calculate your position on Earth. Based on the used frequency, we can expect the accuracy of about three meters under optimal conditions. There are several possible errors that can occur.

There is the multipath error where the signal bounces off of nearby surfaces and the delay where the signal travels through the ionosphere and the troposphere. These errors can be countered by different technologies. Dual frequency GNSS receivers can help to reduce the multipath error by using multiple frequencies at once.

The changing isoniazpheric and tropospheric delay can be compensated by using external correction data. There is no native implementation on smartphones to use this correction data, but there is the Flamingo SDK, which provides such an API, but it's not yet released.

There are many examples of apps that combine AR and GNSS. The most popular is probably Pokemon Go, you see on the left, where you can hunt virtual monsters in the real world. The apps in the middle and on the right are very similar applications.

In the middle is ARGIS Lens and on the right is VGIS. They are used to overlay pipes and electrical wiring into the real world. Creating our own application was straightforward. The digital tier is already a very capable web application we wanted to utilize.

We decided to use Unity on Android to write a simple application. The in-app web browser asset is used to display the web interface. It provides a JavaScript interface that allows us to transfer data between Unity and the web app. The GPS implementation of Unity was not precise enough, so we needed our own GPS plugin to get exact data.

For the AR part, we used the AR plus GPS location asset, which is using Android's ARCore. On the right, you see that the magnetic compass is used by AR plus GPS location plugin. This will be important later on. On the left, you see a screenshot of the map with a placed marker.

And on the right, you see the AR view with the placed object. You see here a view from the top, where the rectangle represents the smartphone and the dotted lines are the possible field of view of the camera. To make it easy to place markers in AR, we implemented a simple interaction.

Where you tap on the smartphone, a ray is shot through the center, and where it hits the detected ground, a marker is placed. Now, if your GPS location is not very precise, the placed marker will have the same offset as the smartphone at the time the marker was created. This offset is along the north-south and east-west axis.

And here we see the problem with the magnetic compass. The compass is required to orient the virtual coordinate system correctly. If the compass is off only by a few degrees, it will result in a large error. The compass is very sensitive and reacts to the magnetic field of internal components and the magnetic field of the earth that depends on your location.

That's why you need to calibrate the compass each time you are using it. Now to our evaluation. We wanted to find out how accurately and precisely a marker can be placed. We had three different locations. One was an open field with an official GPS location marker.

The second was between buildings, also with an official marker on the ground. And the third was a crossroad in the woods with a self-defined position. We measured three distances from two directions and in two different days to have different weather conditions. As the GPS location gets better over time, we took measurements for two minutes.

We used two different devices, a Samsung S10e and a Xiaomi Mi8. The Xiaomi has a dual channel receiver. As references for the compass, we used a good old analog compass. As reference for the direction, the marker on the ground was used.

On the upper right, you see our test setup. We needed to create our own holder from wood and rubber bands. The first metallic prototype interfered with the compass of both smartphones. The compass was calibrated before each measurement. We measured the distance between a marker placed on the coordinates and the reference point.

To see how the compass influences the results, the exact direction from the analog compass was entered as correction value. The reference marker was created by directly standing above the real marker on the ground creating the virtual marker and then moving back to the measurement position. The virtual marker stayed stable on that position.

And here are the results. This was taken at the open field on site 1 from a distance of 10 meters. The figure shows the distribution of the measuring points around the reference, which is located in the origin. The green dots, which represent the deviation without the influence of the compass,

show a relatively small drift of about half a meter along both axes. The distribution can be explained by the drift in positioning by the GNSS. For the black dots, which represent the uncorrected positions of the marker, we see a circular deviation. This compass can be explained by the compass,

which makes the capsule rotate around the position of the smartphone. With the black dots, the drift by GNSS is much more difficult to detect because they are additionally more scattered due to the jitter of the compass. This diagram shows the results at location 3 in the woods.

You can see a drift of about 12 meters along the north-south axis and about 7 meters along the east-west axis. This strong drift could be explained by the multipath error, which is potentially amplified by the surrounding trees at the location.

In this measurement, the green dots, which represent the deviation without the influence of the compass, are so closely grouped that they can hardly be identified as single measurements. The black points, which represent the deviation with the influence of the compass, are clearly divided into three groups, which represent the single measurements in the different distances.

Due to higher angular error in these measurements, the circular path on which the black dots move can be seen very clearly. This behavior is due to jittering of the compass and the fact that the capsule is moved further with constant angular error but increasing distance.

The deviation of GPS coordinates was in 82% of the measurements below the specified 3 meters. The Xiaomi has slightly better results as we expected with the dual-channel GPS. The big problem was the integrated compass. While we have a mean deviation of 3.3 to 6 degrees, which is still not good,

the standard deviation of about 45 degrees means that you cannot rely on the compass at all. I have to remind you that these values are all measured with an already calibrated compass. In conclusion, we can say that the GNSS position was inside expected parameters. The dual-channel GPS had no usable effect in our tests for our application.

The accuracy of the compass could not be measured on the smartphones directly and is the main problem that is even bigger for markers in the distance. We really did not expect this and it makes markers in the distance very unreliable. Augmented reality with ARCore worked very well with some rare exceptions.

To compensate the bad compass, we would like to implement a dynamic compass that is based on the GNSS position to determine the direction. This would require the user to move to get different GNSS locations. This concludes my presentation. Thank you for your attention.

If you have any questions, feel free to write me an email. Thank you and goodbye.