Open Reproducible Research in the Geosciences: Obstacles, Solutions, and Incentives
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 17 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/40497 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
Leibniz MMS Days 20196 / 17
00:00
Presentation of a groupProjective planeDiagramComputer animation
00:16
Presentation of a groupProjective planeLecture/Conference
00:46
Table (information)Directed graphFree groupResultantArithmetic meanComputabilityLecture/ConferenceComputer animation
01:18
Mathematical analysisTerm (mathematics)Theory of relativitySimilarity (geometry)Independence (probability theory)Lecture/Conference
01:35
Table (information)Directed graphComputer animation
01:53
Observational studyFunction (mathematics)Computational physicsProjective planeComputer animation
02:13
Observational studyFunction (mathematics)Projective planeObservational studyMaterialization (paranormal)Local ringMathematical analysisArrow of timeLecture/ConferenceComputer animation
02:55
Observational studyFunction (mathematics)Different (Kate Ryan album)Figurate numberFunction (mathematics)Multiplication signLecture/ConferenceComputer animation
03:28
Observational studyFunction (mathematics)WeightMultiplication signMathematical analysisDirected graphFigurate numberLecture/ConferenceComputer animation
03:55
Observational studyFunction (mathematics)Different (Kate Ryan album)Scaling (geometry)Graph coloringSquare numberSpecial unitary groupDot productLecture/ConferenceComputer animation
04:15
Observational studyFunction (mathematics)Computational physicsNumerical analysisResultantLecture/ConferenceComputer animation
04:30
Phase transitionPhase transitionProjective planeProjektive GeometrieUniverse (mathematics)Computer animation
05:07
Phase transitionUniverse (mathematics)Vapor barrierLecture/ConferenceMeeting/InterviewComputer animation
05:32
Phase transitionMultiplication signMechanism designLecture/ConferenceMeeting/InterviewComputer animation
05:54
IcosahedronPhase transitionMechanism designMaterialization (paranormal)SphereProjective planeResultantMathematical analysisLecture/ConferenceMeeting/InterviewComputer animation
06:37
Mathematical analysisLecture/Conference
06:54
Phase transitionMathematical analysisConnectivity (graph theory)Mathematical analysisComputer animation
07:24
Mathematical analysisProcess (computing)Entire functionLecture/ConferenceMeeting/InterviewComputer animation
07:59
Mathematical analysisPhysicsProbability density functionLink (knot theory)Sheaf (mathematics)Lecture/ConferenceMeeting/InterviewComputer animation
08:29
Process (computing)Open setConnectivity (graph theory)Lecture/ConferenceComputer animation
09:05
Transformation (genetics)OutlierLecture/ConferenceMeeting/Interview
09:28
Process (computing)Entire functionOpen setMathematical analysisComputer animation
10:08
Thermodynamisches SystemFunctional (mathematics)Connectivity (graph theory)Lecture/ConferenceComputer animation
10:39
DisintegrationConnected spaceLie groupConnectivity (graph theory)Moment (mathematics)Computer programmingAdditionLecture/ConferenceMeeting/Interview
11:09
AdditionExtension (kinesiology)Diagram
11:35
Functional (mathematics)Materialization (paranormal)ResultantLecture/ConferenceDiagram
12:02
ResultantDirected graphParameter (computer programming)Function (mathematics)Mathematical analysisLecture/ConferenceDiagram
12:27
Inclusion mapResultantParameter (computer programming)Function (mathematics)MathematicsMathematical analysisMaterialization (paranormal)Multiplication signSet (mathematics)Substitute goodLecture/ConferenceDiagram
12:57
Inclusion mapSubstitute goodMereologyLecture/ConferenceDiagram
13:24
Rule of inferenceLine (geometry)SubsetMereologyEvent horizonTime zoneFunction (mathematics)VelocityVariable (mathematics)Connected spaceSubsetFigurate numberResultantMereologyConnectivity (graph theory)Lecture/ConferenceComputer animation
13:52
WaveletSubsetMereologySlide ruleLine (geometry)ResultantFigurate numberParameter (computer programming)Function (mathematics)Group representationRange (statistics)Slide ruleConfiguration spaceLecture/ConferenceComputer animation
15:09
SubsetMereologyMultiplication signFigurate numberOcean currentFunction (mathematics)Lecture/ConferenceComputer animation
15:29
Process (computing)Electric currentGoodness of fitThermodynamisches SystemComputer animationLecture/Conference
15:47
Process (computing)Electric currentThermodynamisches SystemMathematical analysisAdditionComputer animationLecture/ConferenceMeeting/Interview
16:06
Process (computing)Electric currentOrdinary differential equationPhase transitionStress (mechanics)Statistical hypothesis testingObservational studyOpen setProcess (computing)AdditionPhase transitionMaterialization (paranormal)Projective planeComputer animation
16:39
Phase transitionStatistical hypothesis testingStress (mechanics)Observational studyProjective planeGeometryUniverse (mathematics)Lecture/ConferenceComputer animation
17:02
Phase transitionStress (mechanics)Statistical hypothesis testingObservational studyCartesian coordinate systemPhysical systemLecture/ConferenceMeeting/InterviewComputer animation
17:43
Phase transitionStatistical hypothesis testingStress (mechanics)Observational studyVapor barrierSampling (statistics)Figurate numberLecture/ConferenceMeeting/InterviewComputer animation
18:03
Materialization (paranormal)Process (computing)Order (biology)Lecture/Conference
18:19
Phase transitionStatistical hypothesis testingStress (mechanics)Observational studyMultiplication signStatistical hypothesis testingStress (mechanics)Computer animationLecture/ConferenceMeeting/Interview
18:38
Phase transitionStatistical hypothesis testingStress (mechanics)Observational studyAutoregressive conditional heteroskedasticityLogarithmPhysical lawMultiplication signComputer animation
18:55
Open setPhase transitionStatistical hypothesis testingStress (mechanics)Observational studyMany-sorted logicObservational studyMultiplication signLecture/ConferenceComputer animation
19:18
Data analysisMathematical analysisParameter (computer programming)Process (computing)Phase transitionStatistical hypothesis testingStress (mechanics)Observational studyConnected spaceProcess (computing)Group actionComputer animation
19:55
Inverse elementConnected spaceProcess (computing)Mathematical analysisDiameterLecture/ConferenceComputer animation
20:11
Multiplication signLecture/Conference
Transcript: English(auto-generated)
00:00
My name is Marcus, I'm from the Institute for Geoinformatics at the University of Münster. Today I would like to talk a bit about our project O2R, which stands for Opening Reproducible Research. So I think it's a good idea to start at the beginning with some kind of an agenda. So at the beginning of my talk I would like to talk about reproducibility in general.
00:21
So if there is a problem, just a little spoiler, yes there is a problem. That's why I'm here, that's why I'm working at this project. At this O2R project. Afterwards I would like to talk a bit about our core concept, which is the executable research compendium. And what you can get out of it if you use it.
00:41
And finally a brief overview of our goals in our follow-up project O2R2. I will not be able to go into detail with all these topics, but feel free to ask any questions at the end of this talk. Or after this talk or tonight at a conference dinner or via Twitter, GitHub or email. Feel free.
01:00
So let's start with a quick definition of what we understand reproducibility to mean. So with reproducibility we mean computational reproducibility. Meaning that you can achieve the same results as reported in the paper using the same source code files and the same data files. In my experience there is usually some confusion with related terms.
01:21
For example replicability, which means that you can come to similar conclusions with independent experiments. So you collect new data, you implement a new analysis and then you achieve similar conclusions. I wouldn't say that reproducibility is more important than replicability.
01:40
Some would even argue that replicability is actually more important than reproducibility. But I hope that I can convince you a bit that reproducibility is also very important for scientific work. In our project we focus on the geosciences. For example landscape ecology, geography, planetology, geochemistry.
02:04
So disciplines like that. But I hope that if you work with data and code that you can also take home some of the messages that I have for you. So at the beginning of our project we ran a little reproducibility study. So I tried to find papers that had code and data attached.
02:21
And this was already the first issue. So most of the articles that I looked at didn't have any code, any data, they had nothing. Okay, the first issue is lack of materials. But at the end I found some papers that had our code attached and data and I tried to execute the analysis. And this is already the second issue. Most of the papers that had code attached are not executable.
02:43
So I tried to open the files in my local programming environment. I tried to run the code and what happened, I had a lot of errors, a lot of technical issues. For example the library was deprecated, the file directories were different. So they were for the computer of the original analyst and not general I would say.
03:05
And yeah, so this is the second issue, that most papers are not executable. After I got some papers running I looked at the output, mainly at the figures. And I compared these figures to those figures in the original publication.
03:22
And this is the third issue. So most of the output was different from what was reported in the papers. I think this is a good time to emphasize that I don't want to blame the original authors. I just want to encourage them to publish code and data in an executable and reproducible way. So here I have two examples from a geoscientific paper.
03:44
At the top map you see the figure from the paper. This is the original one. And here you can see what I got out of the source code when I reproduced or re-executed the analysis. So there are a couple of differences. For example the background map is slightly different, the colors are a bit different, the scale is a bit different.
04:03
So these are rather design related issues. You could argue that these are not so dramatic. But let's have a look at this green square. There are two dots. And these two dots are not available in the original map. So the actual results, the numbers are also different. And this is a rather serious issue I would say.
04:24
So if you come back to the question, is there a lack of reproducibility? Is it a big problem? Yes, it is. But there is a couple of research going on on this topic on reproducible research. One of these projects is our project O2R.
04:41
We had a first phase which was a two-year project starting in 2016. We were three researchers. So Marc Schutz, actually from the University Library of Münster. And my colleague Daniel Nist who is also from the Institute for Geoinformatics. The first phase recently came to an end. But we asked for a follow-up grant and we got that accepted.
05:03
So we have two and a half more years to do some additional work. But I will come to that at the end of this talk. As I already mentioned, it's a collaboration between the University Library, also of University Library of Münster. With the Institute of Geoinformatics and publishers. So we work with, for example with Copernicus and SCB.
05:23
We have a couple of goals. First of all, we would like to identify the barriers. The barriers that prevent geoscientists from publishing open reproducible research. So I already mentioned some technical issues. But there are also some cultural issues. So-called cultural issues. So for example, there's a lack of time and lack of incentives for preparing code and data for publication.
05:47
Another problem is the lack of knowledge. So most people don't know actually when they achieved reproducible research. So when they upload their supplemental materials, there's no mechanism to check if the supplements are executable and if they are reproducible.
06:02
And finally there's also something that is called the fear of being scooped. Which means that other researchers take the data, take the code and publish a paper that the original author actually wanted to publish. It doesn't happen very often, so I would say this fear is a bit abstract.
06:21
But still very common. In our project we would like to design and evaluate ways to overcome these barriers. And we would like to develop an approach to reap the benefits of reproducible research. So let's be honest, just being able to execute the analysis and achieving the same results is nice.
06:42
But there are a lot of other and further opportunities that we get if data is available and if the source code is available in an executable way. And finally we want to implement a platform that realizes the approach and to test it with real users. So we have a rather user-centered approach.
07:01
These are the basic ideas. So first of all we want to provide an easy way to publish the paper, the data and the analysis together. Usually these three components are completely separated and disconnected. And we want to develop an approach that allows authors to easily publish these things together.
07:21
Then we want to integrate this with existing publication procedures. So we also have to be realistic. There are publishers who have their own publication processes. And we somehow have to be able to integrate our things into their infrastructure. They will not change the entire infrastructure for our technologies.
07:41
And finally we want to investigate potential incentives which might motivate authors and readers to publish open reproducible research and to use it. The core concept in our research is the executable research compendium. There are two ways for the executable research compendium to be published. One way is to replace the PDF file entirely.
08:03
So researchers would not publish or submit PDF files but executable research compendium. This is rather disruptive. But there is another opportunity for example to use executable research compendium as supplemental material. So now if you have a PDF you scroll down to the supplemental material section.
08:23
And then you have a link which brings you to the executable research compendium. And not to a folder which contains the data and the code. And where you have to find out by yourself how to run it and how to modify things. And if we have that this paves the way for new opportunities and possibilities that might motivate authors to publish that.
08:43
And this is how it looks like. This is the executable research compendium. Some of you have seen that yesterday at my poster. But just a quick repetition. The executable research compendium has four components. Which is the data. Ideally submitted as raw data. So what we usually suggest to researchers is to stop doing manual data processing.
09:05
So you shouldn't go to your Excel file and delete the outliers by yourself. Or manipulate the data by yourself doing data transformations. We rather suggest doing that in a script. Because in six months if you look at your data you might not remember what you did with the data.
09:20
You might not know why you deleted something or why you changed something. And this is also difficult for others to follow how you did something. How you did the data processing. And it should be included as a file and as open access of course. Because we rather focus on open technologies and suggest using them to submit ERCs.
09:43
Then we have the software. The software comprises the source code scripts. Which should be also open source. And we encapsulate all that stuff in a docker container. To make it very easily runnable. And if we have that in an executable way we can at the end provide a button basically.
10:03
Where you can click and rerun the entire analysis. Just with a single click. You don't have to download the files. You don't have to open them in your local environment. You don't have to find out how to start and run it. You can just click on a single button and that's it. Then we have the documentation. The documentation includes the actual article.
10:22
So the paper. Some instructions on how to run the code. And metadata. So metadata are very interesting with respect to software and data. So these two components include a lot of interesting information we can search for. For example libraries, functionalities and all that stuff.
10:41
And finally we have something that is called UI bindings. UI bindings link the three components. So data, software and documentation. So we provide a way to link the text in the paper to the source code and the data. To avoid that these three components are completely disconnected or separated. And therefore we have UI widgets.
11:02
Which help users to interact with papers without additional programming. I will come to that in a moment. So now if we have code and data available in an accessible and executable way we get a lot of additional opportunities. And these opportunities are summarized in this extended workflow for readers.
11:22
This is not only for readers but also interesting for authors. So let's start with the first step, discovery. If we have data and code accessible and executable we have additional opportunities to search for papers. We can search for papers that use a certain functionality from a certain library implemented in a certain programming language.
11:43
And this is also interesting for authors. They can make the work much better findable to get citations at the end. The next step, inspection. Then readers and reviewers can look at the materials that were used to produce the results from the paper.
12:01
And authors can easily convey how they achieved the results from the paper. So it's also sometimes difficult for authors to explain how they did something. They can just refer to the source code, to the data. And then explain how they did that. And the next step, manipulation. Many papers have some parameters in the analysis.
12:24
And it would be interesting to know how these parameters affect the final output. So if the output changes, if you change the parameters. So what happens then? This is interesting for readers and reviewers but also for authors to show that the results are either robust or fragile.
12:41
Finally, substitution. Substitution means that you can substitute materials. For example, the dataset. By an own dataset. If you have another one. Or you can substitute the analysis if you want to. I already heard a couple of times that you like the FAIR principles. And this is also what you can find here in this workflow.
13:02
So we have discovery, which refers to findability. Inspection, which refers to accessibility. Reusability, which refers to manipulation and substitution. And interoperability is in part covered by substitution. And of course we also argue that each of these steps also contribute to a reader's understanding when reading a scientific paper.
13:29
Coming back to the UI bindings. You remember the component of the ERC, of the executable research companion, the bindings. Bindings connect those parts of the R script and data subsets that we use to compute a specific computational result.
13:44
Let's say Figure 1 in the paper. This is also what I presented yesterday on the poster. So let's say we want to create an interactive figure for Figure 1 in a paper. The first step would be to specify the result, which is Figure 1. The second step would be to select all the source code lines that are needed to run Figure 1.
14:05
In the second step, it is required to specify the parameter that should be interactive. Because the author says, OK, this parameter I set to 2.0. And now I would like to make it a bit more interactive to allow some other values.
14:24
And this can be done by configuring a user interface widget. In this case it's a slider with a range from 0.1 to 3.5. And it can be changed with a step size 0.1. Here's a small explanation that explains what the UI widget actually does.
14:43
And that's it. This is all we need to create an interactive figure. And this is how it looks like. On the left side, we still have the static representation of the paper with the static figure. And on the right side, we can manipulate the figure. So this is the slider that we configured in the previous step.
15:03
And if we change the value of the slider, then we can see immediately the output on the right side. Immediately it's true for this example. But in other examples, of course, it takes a bit of time. Maybe a few minutes to recompute the figure with the new value.
15:21
These are some special cases where we would need some further development. For example, pre-rendering the output. So let's compare executable research compendia versus current practice, which is PDF files.
15:40
Executable research compendia offers, for example, one-click reproduce. So it's not anymore needed to download all that stuff, to run it in a local environment. And to find out how to run it, you can just click on a button and reproduce the analysis. It opens everything, so you can look at all the data, all the code.
16:01
And it's completely transparent, nothing is hidden. And you have additional opportunities for reuse. So you have some new interaction possibilities. You can manipulate things, you can look at the materials, and so on. But it's also important to integrate ESCs in the existing scientific process.
16:23
So this is what I mentioned at the beginning, when I talked about the publishers. So we have to think further than just defining and implementing ESCs. And this is what we would like to do in the second phase of our project.
16:41
So in our O2R2 project. We have another two and a half years to work on that problem. Now we have two researchers, not three anymore. There is still a collaboration between the University Library of Münster and the Institute for Geoinformatics and the publishers Copernicus Publications and LCV.
17:03
We have a couple of new goals. So we want to implement some pilot applications. So we want to integrate our concepts and technologies and tools and features into the infrastructure of the publishers. And this is an interesting step, I think. So we can be sure that the publishers don't want to change their whole infrastructure.
17:25
Just to include our services. But they are interested in these things, in this reproducibility thing and interactivity thing. And yeah, we have to find out how we can integrate our things into their submission system.
17:45
There are still some barriers that we have to eliminate for authors, readers, reviewers and publishers. So authors need easy ways to create these ESCs and particularly the interactive figures. We have to provide low effort features for readers to use executable research compendia.
18:04
And we want to find out how the reading process affects the understanding of the reader. Then we have reviewers. Reviewers previously looked at the PDF file, maybe at the supplemental materials and now they have more possibilities. But it might be that it also takes a bit more time.
18:21
So we have to investigate these issues as well. And finally, the publishers. So how can we integrate our technologies into the infrastructure of the publishers? Finally, evaluation. We want to evaluate a couple of aspects. So first of all, the technical aspects or stress tests, for example. What happens if 100 readers try to reproduce a paper at the same time, for example.
18:47
Then we have some user aspects. For example, logging. We want to log how users use the platform and the interactive tools. We also want to provide some sort of support for authors to create executable research compendia and for readers to use it.
19:03
And we want to run some user studies to evaluate usability, user experience and stuff like that. Do I still have some time or four minutes? Including discussion. Including discussion. Okay.
19:22
Maybe just a quick run through. What I mentioned here is that we want to integrate our ERC into the infrastructure of the publishers. This is the first idea that we have with the publication process. It starts with the author who creates an unvalidated research compendium, which is this one here.
19:41
So it is basically the workspace, including the R scripts and the data, which then becomes an executable research compendium on our platform. When it is reviewed, it becomes a reviewed executable research compendium. And if it gets accepted, then it becomes a published executable research compendium, which can then be used by other readers to create new executable research compendium.
20:04
Okay, so I am open for any questions or comments. Thank you very much. There is time for discussion. Thank you very much.