So, I'll talk to you a little bit about reproducibility and so I approached it openness from the perspective of reproducibility or replication.

And I'll try to give an overview of maybe what are the needs and what does it mean, at least from my perspective and what is it in our field a little bit. So, let's move first with a little introduction so I'm currently directing an institute called Institute of Interactive

Technologies, and this is a computer science department I switched from geography, three years ago to this group. And, and we deal a lot with human computer interaction and extended reality and also through my past life, which I will have a small slide as well.

I've done a lot of empirical studies with visualization, and some strong interest with 3d, which includes virtual reality, partly because my PhD is in photogrammetry and my minor was cartography so I'm kind of in both worlds. And I also take some active roles in these communities. And just so those of you who don't know me.

You can see that I actually have a number of hats that enables me to observe in actually in two communities at least two of the sub communities, including computer science, maybe one can say three communities.

I also end up going to psychology meeting sometimes because of that. I've done a number of visualization projects, I'm not able to give you a lot of examples from each of them, but just wanted to show you the range. So a lot of the visualization projects that one of them was about shadows and color, how to make 3d

visualization through shadows, or how to make best use of color or how about discriminate color, or what is the level of appropriate realism or how do we measure visual complexity, how do we design interfaces, how do you deal with 3d and information visualization. How do we do gaze adapted displays because I do a lot of eye tracking as well. And then I did a

really large number of where I still actively do not large number of extended reality stuff for like collaborative extended reality, virtual reality as a lab environment for education. There you have for example like how do you do

science experiments in VR, and sometimes for capturing age related cognitive issues because spatial navigation is related. And this is kind of a huge range of things, but what is in common is that there's always an element of visualization interaction and eye tracking, it's like as a bonus on top of it because it's an area that I have a lot of.

I guess background in, and then in every project disappear. Of course the dramatic focus moves around but methodologically these are what it is. So I also want to upfront share acknowledgments because everything I've done and said today, also have been informed by

a number of people that I closely collaborate with publish with and I didn't even list my collaborators these are actually team members who actively contributed either in the past or are still contributing, I'll show two examples, and then I'll highlight from each of these two people.

Okay, let's jump into the current state of augmented reality or virtual reality or extended reality, let's have a look at this little video. I like showing this one. You might remember having a desk, where you had I'll play it again, where you had a fax machine, maybe a calendar on your wall maybe

a Rolodex for the addresses, maybe a camera to take pictures and the phone to make calls maybe an address book, and so on and so

And what is perhaps quite nice is that virtual reality is the glasses so the chances are the glass will eat the computer, at least that is direction that industry is pushing at the moment. The next big thing might be the variable, rather than hand controlled. And that creates all kinds of.

There's all kinds of motivations for it. And one example that I also like to show I'll just play this one. Can you hear the sound. I think maybe you don't hear the sound Can I change that I think it shares sound yeah this is I think important.

Let me just, it's, I'll play it on like few seconds. So the point that I want to make with this one is it's a collection of it's a fairly long video collection of virtual experiences, and it has quite a few nice ones with gravity gravity experience of being on a chair and so on.

And this one is a roller coaster experience. One can also flying and things like this, but what what wanted to what I wanted to say is that the screen. It suggests that the emotional response to the experience is rather real so the fear that you feel from that experience is

very ill. It's a virtual experience, and you know it, but the emotional response is just the same right so this is where you have a very interesting occasions.

So, there's also this one some of you may know that it was I think first interaction farm and then now it's in a Turkish farm. Some people seem to put, we are headsets on cows, and there's a. Apparently, it increases me feel that that is very interesting they show them like green pastures, the fields during the winter months or

so. And then if you do that, if you really get more meat because you do that, which means that it's not only psychological but also physiological that you have responses. And I think there's other evidence for it as well, because in some

fields, like in medicine, there's even attempts to try to see if we are can be used for pain management. And that's a bit of a psychophysiological mix. Of course, you can also do things like this which is something that we also published with Alex but it's like a historical walkthrough I think many of you have seen potentially have some similar examples I find it quite inspiring.

Quite like putting the stuff in their spatial context and you know what it was here before, but it also means I mean this this example is also quite nice because what it does is that it shows you what it was in the past within this portal, but it also includes

what it is now. And this has a lot of implications, potentially. So, if you think of this coming developments with the virtual reality and extended reality. If you take an optimistic view, which I tend to rather rely but I hope so but so you can have wonderful experiences you can get this floating in the space or get new insights that

you couldn't get otherwise, we can't easily go to Mars, but we can walk on on Mars in a virtual environment, or we can go under the oceans or whatever some, some things are prohibitive or very expensive or impossible but we can make that happen. And that is what's what's what's the beauty of it right so you can potentially take a walk in your grandparents shoes.

And if you do that, maybe it creates more empathy, you can potentially cure diseases, you can generate a lot of fun. So there's a lot of things to be said, and for science. It also gives very interesting

new opportunities, because you can do controlled experiments, which you can't do in the real world because there. There's a lot of uncontrolled variables, but in VR you can exactly get everything right, the lighting the temperature, whatever you want. And then there's a number of things that you can then experiment so which variable causes what.

So there's a lot of things that are attractive about it. Of course the dystopian view of it is, if you are familiar with the TV show black merits exactly in this direction. So, this. Remember this historical walk example. So the world your world might be controlled by someone else. So if you are wearing a glass or a variable other variable

a chip, which at the moment they are cumbersome, but I'm thinking futuristic Lee, that it might eventually get lighter because they are really working on it, and Microsoft is working on it and Facebook is working on it and Apple is working on it and at some point I think they will get there, that they are perhaps light enough for us to, to imagine being able to use it.

But as soon as you have that of course it can be hacked to, it can be controlled by others and the information might be pushed by others. And of course there's huge implications about extremely technology driven world which is, you know, already the case it's not like

it's not the case now. But if you have, if you have tech abilities, you are basically joining a class of overlords that you can control stuff but people don't, then they have a problem, either they don't have enough opportunity for education, or they don't have.

I don't know, because of the aging or other cognitive abilities or because of their professional focus. It's possible that they are not able to get as much from these devices as it is with the smartphones. And so, and nightmare. Of course, I can totally see that happening it's already in every possible place that we digitally. And

of course there's all kinds of unfortunately very sad possibilities of abuse and, and of course also existential shift in the sense that, for example, object permanence might be removed so the objects that if you can't tell what is virtual and what is real normally

humans developmentally rely on objects being permanent you know you close your eyes you open your eyes to objects are going to be there. And that is something very fundamental for us to orient ourselves and then if you start removing that from children's development, you might have a very interesting very different kind of development, which is unknown.

And it's very interesting implications. Okay, so that's kind of the opening spiel, but the question was whether we could open data could help against some of these dystopian outcomes, whether we could make sure that things are documented things are fairly treated things

are made available the process in between, is actually enforced so that at least those who can has have the opportunity to double check what's going on with their world. So the replication crisis I think some of you may have covered it during the conference I'm going to

say a few words about that as well, is real, and this is also something open data would enable. And I think that's also important for against this dystopian view because there you can also say that okay but what is really happening can I replicate what happened so it enables you to understand.

And this is not so different for human computer interaction, especially the virtual reality. And this message to in 2021 paper that it is similar to vulnerable similarly to the same challenges. So of course I mean this is something that is something that we

can all easily stand behind. I mean reproducibility rigor transparency independent verification are cornerstones of the scientific method, we can't not have that we must have that otherwise we are not doing science, and like one thing that perhaps at this point because we talked about here already reproducibility.

And one thing that I thought that might be also worth taking a moment is to distinguish reproducibility replication. I think, as you many of you again is probably well aware, people use this to interchangeably. If you wanted to be, I

also confuse them. But if you wanted to be correct, I think the dictionary definitions are so that reproducibility is that you can actually come back and reproduce your own results so you can repeat the analysis and get the same results so that the data is there. The steps or any data pre processing any analysis steps that is well documented so you can actually you yourself 10 years later, could run the analysis

again. So it's the software did not expire, and so on and so forth so that you could actually come. Exactly. So this is the reproducibility replication is rather. Okay, so I have this one study I observed something, if I do it with another population or if I do it again

in using something else or just someone else does exactly the same. Could they arrive at the same result so that that's the distinction between these two terms. So, here I think in this, in this one, actually, is used exactly correctly, the reproducibility isn't to protect all the processes

all the steps that are applied in the process the data and software and documented well so that you can actually come back and do it yourself or someone else can reproduce. Alright, and yeah for our case, there's also some distinction. There's experimental research

and other kinds of research, other kinds of empirical research if you like. So for example if you do spatial analysis, and you're sampling matches. So you have to be very explicit about how you documented and someone else, or if you want to do it again, you would have to sample, maybe another at another time, a time point right

so you have to document things carefully so that it's actually justifiable, and maintaining the data and rotating documenting coding etc. And then the experimental work and by experimental work, I mean actually human experiments in this case but of course experiments

could be also completely computational here, but also matches this to really document the characteristics of the participants, and that the context in which the data was collected as well described. And this this need to be documented in great detail, if you want to

do it correctly, because each variable can actually affect the outcome and it does. In, in sometimes subtle things affect the outcome, like for example forms of communication priming, you know, if participants come and to some of them, you say yes, don't worry it's an easy experiment or something and the others you don't say that this

apparently affects people's confidence which affects their performance. So there's all kinds of like tiny on, like for unfamiliar I it looks unimportant details that might actually change the outcomes of your observations, and all of that is very important

to document because someone else would want to then reproduce what you did may not be able to do it, unless you provide them such detail. Excuse me. So, at some point, nature pose the question to scientists and directly asked is there a reproducibility crisis and 52% of the people who joined said yes, there's a significant crisis.

There was 1500 people who joined and 38% said yes, a slight crisis, and only very small percent say there isn't one. I guess, this, I don't know what kind of research to these people do, and then 7

% who said they weren't sure but but is, of course, very striking is that more than half. And given the other 40% also is on that side, but people are aware that there's a serious problem going on. And there's a lot of efforts trying to demonstrate what happened in this particular piece, they have tried to replicate

psychology studies, and they were able to do somewhere in the range of 36 to 47% so less than half in the positive outcome. And that is what created kind of a huge base already from 2015 or a little bit earlier even.

So this one other little thing to remember is that the criteria for what reproducibility should be is different. I actually found some taxonomies,

but I didn't want to turn that into a full lecture given that I only have 30 minutes but I'm happy to provide you some. There's some people who try to taxonomies approaches to reproducibility, and that does indeed very in some ways so people try to now like this 59 different definitions or criteria developed by different people.

So, in any case, that said, that noted on one side, we have seen evidence and I'm going to go through this a little bit quicker estimating in psychology, cancer research, computer systems research is just this three examples here shows you why there's such a big difference

in what reproducibility means because this actually are very different disciplines, maybe with different needs. And similarly, you can see that for computational and biology computer science biology. And I think I'm trying to put you. Okay, geoscience and statistics, chemistry, social sciences, and then we also

have a bunch of efforts from our corner, you have in VGI, and you have some GI science, some.

Some of you might be even in the audience if they are, if you are there. Hi, and you can you have a number of publications coming up, trying to propose ways to deal with reproducibility. We have not seen much for empirical studies in cartography yet. There was an effort in almost a couple of years ago with International Cartographic Association, led by Amy, this is a little bit open we have all contributed a little bit and there may

be some still something that comes out of it but at the moment there. So, just, I have 10 minutes left I'm trying to monitor the time. I'll give you one or two examples of like trying to replicate something on our side in this one study with shadows,

we tried to be done a lot of studies on this. Some of you may not be familiar, and some of you probably are very familiar with this now, but depending on where the light source is, one can perceive a shape convex or concave.

So here the light is coming from below and here you can see that shadow is up here and light is up there right so you can see that the. Maybe if there was a light source that is from, and then if you rotate it around this is hundred 80 degrees rotated, then the

light is now up here, and then the shape for the majority of people, a good 99% of the people, then the shape actually changes from convex to concave. This is everything is diverted here if you take the time to study this is called light from above prior it's a cognitive

bias that creates an illusion, mainly because the sun is up there and state lights are up there and ceiling lights are up there and the moon is up there and most of the time in your life. You have millions of experiences where shadows are below because light is above, and then the mind uses a pre categorization, rather than interpreting the perceptual signal, it uses a previous knowledge

and interviews with the perception. This happens, also in satellite images. We studied how prevalent this is. It happens in shaded relief maps so if you put the light source in the wrong place, people will perceive valleys as religious and religious as balance.

Here, this is the rotated version, you might see it from here from the Google's head. If you show this right side up the valley looks like a rich and snow looks like it's in the valley floors, and here this is actually a valley and not a rich because the light comes from the southeast.

And this happens in the northern hemisphere images as a side note, because the satellites are sun synchronous and when the pictures are taken the tilt of the earth and the position of the sun creates this issue that the northern slopes are in shade and the southern

slopes are eliminated. And this is what creates a little bit of a then you show the image North up then creates this illusion. So in this study in the original study, we have a cartographers notice by the way, and the convention is to put the light source at 315 year.

And in this study what we did is we took the shaded relief maps like this one, we generated them. It's harder to do it with the satellites this way. And then we say that okay let's just do some stepwise illumination and see what happens. And then we asked people, you know, is that what, like, clearly a valley or clearly a rich and then they just go ahead

and they mark what they think they are looking at, and everything blue marks that it is indeed a value if they are marking a value and everything that says they are marking a value but it's actually a rich. So you can see that the southern angles, produce a very strong illusion northern angles are overall better, and then this like sides are a little bit more ambivalent

so here it's almost half and half. And so, this study showed us that actually 337 this this was a little bit better than the cartography convention, we published this, it was received well, and the northern angle here was even a little better.

But then there's something called the left by is not only left from above but also above left and that you can also see it here. So, instead of getting the light complete up. It's, it's somehow better for our mind or we assume we learn that shadows are actually in a particular place that light is rather on the above left.

This has been documented also in perception psychology before, and people speculate that it is to do with being right handed mostly, and you have some tendency of orienting yourself if you do manual work, you have a tendency of orienting yourself against

the light so that the light comes from the right and the architects will build the schools like that that the windows are on our left side, and this means that you don't get shadow of your hands on whatever you're doing, which apparently in some cases

can be an obstacle. So this is I'm saying this because we wanted to replicate the study, and it will have some relevance to that so we wanted to see. There was some questions about different hemispheres people we've been, and I collaborate with South African sometimes in Australia, so I ended

up trying to replicate this, but very briefly, with this left bias still for us it was at the 13.4 degrees in our study in our original 2016 study and others have documented this number differently. So somebody said it's at the minus 22 minus 24 that's left, and for for the some studies, they say that it is actually.

So, these are not mine since they are just two degrees. The minuses are actually bullet points. So 22 degrees for this guy's they have only seven people, and here we have a different one for the left handed and

the right handed one. And here again you have different for the left handed and the right handed people, and the difference is quite high. So, this is something that comes interesting because in our original study we had only one left handed person we did ask the handedness but we didn't try to recruit them half and half.

And this is what you got here we have the, this is the replication effort in South Africa, so you can basically see that it's still like the highest accuracy with the terrain forms are here in this case. And then it gets a little bit worse with them up northern angle so this is zero degrees north, and in the

application, the study the same angle it was exactly the same study, the same angle, but the north, the numbers are actually identical. And this is something that we published. But here we have a little bit more like we have three people are left handed and seven people who call themselves both handed so ambidextrous so they are.

That's what they're saying so what we did is we also went ahead and recruited a bit more balanced sample, and we have now put 14 people out of 35 left handed. And now you can see that the that the accuracy actually shifts a little bit more towards us.

And this, if you take this I did not allow these are because these are kind of in progress. I did not calculate the left bias, exactly, but this is linked, you will see that with the left handed people the left bias is there, but it is closer to the north, it will be what we will see. And that was, for example, the replication the opening the beta comparing it with some other team or

replicating our own work in this case, and that made us realize that maybe handedness has an important. It plays an important role here. Another example I mean I can talk about this for long but I need to

wrap it up. Another example very quickly, is that something that we also recently published. It's, there's something called mental rotations test. Some of you may be familiar with that it's a bunch of cubes that you are supposed to mentally rotate, and that's a frequently used study for measuring spatial abilities, and we wanted to replicate.

If 2d MRT would be actually different if you showed it in stereo 3d, so you still can't rotate the object, mental, you still have to mentally rotate them, but the shapes are 3d rather than 2d, because perception is speaking, and there we have been able to replicate

the two definings fully, and then some observation by Peters said that people actually, there's a gender gap in this and Peters documented that when you show them in stereo the gender gap narrows, and we also replicated that so moving from 2d to stereo 3d

improves everybody's performance with the MRT, it's interesting because it's. I think it reduces the fortress like you don't have to imagine it in 3d so there's two aspects but it's visualizing in your head what the shape looks like, and the address to rotate it

and there's actually two steps I would say and then by showing it in stereo, you are actually making it easier for people to imagine the shapes in 3d. And here again we can we could show that the gender gap narrows here the conditions have changed a lot it was not so easy to replicate

this based on someone else's study based on a publication alone just wanted to also mention that. But of course it is very important and valuable because if you keep doing this repeated observations, you learn what you learn is potentially eventually more generalizable. So, someone else's doing some work in replication in XR I just wanted to mention that there's actually an ongoing effort that

are limited there's limited number of replication studies so this person is a PhD student in the US, I'm not collaborating with them but I know some activities by them. So, he was narrowing, and how many replication papers he could find

found only five in augmented reality and only two in virtual reality that is that includes actually empirical work. All right, I'm almost done. So this is their effort. This is the I've been on a towel. That's actually active they are doing a second workshop if you are interested if you are working on this domain, and they want to

discuss the replication crisis in empirical science, and especially in XR as you see them extend augmented reality, and they have an open call. So, and the paper submission is actually first of all, and it's reasonable I guess if you have something to say, and you're well, that would be perhaps welcome by them.

I contacted them and, and they are keen to get contributions. Just, it's Mars, if you are not familiar, it's also a premium conference for XR. All right, so I'm wrapping up. So, we have some. So, what is sort of like challenges and opportunities if you like, there is some opportunities, because now grant agencies are pushing

for open at us pushing for it. Swiss SNF is pushing for it and they provide some money as well. And we have these events like these events like this one that we are in mass like this. So there's also some individual

initiatives and and workshops and there's some shift in the culture, but which, which is I think wonderful, but there's also of course, you know, the grants are super important because the way that, for example, I have not opened a lot of my data yet. There's one

reason that that of course is that one needs time to properly do it, you need to annotated you need to prepare it for someone else to actually use it is restoring it somewhere, not a big problem, but to be able to really open it, you also need to really prepare the data, and that's not an insignificant effort.

And this is why I think the money is important that this is budgeted in every project. And, but also, there's a sort of a strange side effect to this because if you have a lot of institutions saying that out okay we will give you money if you publish

open access will give you money if you get some commercially minded individuals jumping or and they try to exploit the system so there's already some unfortunately corrupt behaviors are very suspicious behaviors by some open, open access journals which which is, which is a little bit complicated and good to be aware of. All right, I think I'm going to stop here.

Thank you very much. Just wanted to finish with a positive message that the size isn't broken it's, yes, there's a lot of challenges. It's just harder it doesn't application doesn't mean it's wrong what was observed as such, but

it's just that we're variables are important and we should just be careful about generalizing them. Thank you very much.