Preregistration of research for theses - a new standard?
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 30 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/58579 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
6
00:00
Computer animation
07:13
Computer animation
14:26
Computer animation
18:53
Computer animation
Transcript: English(auto-generated)
00:01
Good morning, afternoon or evening. Thank you for listening to my talk about preregistration research for thesis. My name is Teresa Shimula and I will be your guide in this video presentation. In this talk I will talk about the importance of principles of openness and transparency in research. In particular
00:20
I will focus on one of the approaches, namely preregistration of research. Preregistration of research is now a common part of a good scientific practice registering hypothesis, methods, analysis, techniques, dates and so on before a study begins help to make a more
00:42
robust, credible transparent and open research. If preregistration has so many advantages, why is not yet a common part of writing thesis? Make yourself comfortable and learn about how preregistration could become the new standard in the thesis writing.
01:04
As many of you would know the science is now in a reproducibility or sometimes called replication crisis. Reproducibility crisis is ongoing methodological crisis whereas a difficult or even impossible to reproduce the results of many scientific studies.
01:21
Because the reproducibility of scientific results is an essential part of science, such failures undermine the credibility of theories and the science itself. There are a lot of things that contribute to this but in general a lot of these issues come down to the hypothetical deductive model of science being short-circuited by a questionable research practice. On a slide
01:46
you can see a good illustration that has been created by the Center of Open Science. Lack of replication, low statistical power, p-hacking, harking, publication bias and lack of data. All these questionable practices
02:01
have contributes to creating a reproducible crisis. So how can we strengthen the reproducibility of science? One possible answer is to preregister research. So what is preregistration? In simple terms
02:22
we can say that preregistration is the process of documenting a research plan that is done at the beginning of a research process, usually before data collection. We can also say that preregistration involves creating permanent records of your study plans before you look at the data. There are many definitions of preregistration, but what is important is that there are some elements that most definitions have in common.
02:47
Personally, I like the view of preregistration as a kind of roadmap for research, which is data-stamped, frozen, read-only research plan, which in most case is created before the study itself and it's published in a public repository. Of course, there are exceptions, such there is for example
03:07
embargo on the preregistration or the preregistration is done after the data collection, but in a general sense we can define preregistration as I state here. What is important is that that preregistration helps the researcher prove
03:23
which hypotheses were actually confirmatory by date-stamping them. Preregistration also offers the researcher the opportunity to front-load the writing process and reduce the problems associated with miscommunication between collaborators. Also, preregistration helps to avoid later criticism on
03:44
temptation to look for reasons to remove data that do not fit in the hypothesis. There are different types of preregistration. For classic preregistration, also known as
04:00
unreviewed preregistration, the researcher creates a detailed description of her or his plans for a study as possible and save those plans in a time-stanned uneditable document at some public repository. These records can be later shared with reviewers, editors or some other researchers.
04:24
Second type of preregistration is so-called register reports, also known as reviewed preregistration. In this case, researchers submit a detailed proposal for a study to a journal before conducting the real study. These registered reports have the same virtues as
04:42
preregistration, but they also address the problem of publication bias because the studies are published regardless the outcomes of the studies. Some journals require this registered reporting. For example, here you can see two-stage model of publication process from one journal from PLOS One. In practice, you
05:05
publish the peer-reviewed preregistration first, then you make your analysis, write a report and then you write the actual publication regardless the outcomes of your publication. This mechanism fosters transparency and reporting in publishing.
05:24
And the last type of preregistration is register replication report. A variant of registered replication reports focus on a direct replication of one or more original studies, original findings. Many labs follow the same
05:42
preregister plan and the results from all these independent studies from all the independent labs are published collectively regardless of outcomes of the individual studies. In this presentation, and given the topic of my talk, I always going to talk about the first type of
06:03
preregistration, the one that didn't go through the peer review process. Let's now take a look what such a preregistration should contain. Well, of course, it depends on the field of study. Also, it depends on your personal decision, but it is possible to generalize the content of the preregistration.
06:24
In this respect, I agree with the guidance provided by the Open Science Framework. Their simple web application guides the user through creating a preregistration in few simple steps as you can see in this video. First, you need to provide information about the study itself. Things like
06:44
title of the study, authors, pre-description of your study and specification of your hypothesis. In the second step, you are asked to describe the overall design of the study. Here, for example, you should define what type of study it is, whatever it is. It will be an experiment,
07:03
observatory study or some meta-analysis or some completely other type. Then you describe who is aware of this experimental manipulation within a study. You must also describe the study design itself. And then you get to the step three where you are describing how you plan to collect samples.
07:25
Wherever you will use data that already exists or you will collect your own. You shouldn't forget to describe the sample size. In the variables section, you describe both manipulated and measured variables that will be later used in your confirmatory
07:41
analysis plan. In the last step, you will describe your analysis plan, for example statistical models or how you will deal with the possible exclusion or missing data. As you can see on the slide and as I mentioned, the Open Science Framework offers a great step-by-step guidelines that will walk you through
08:02
creation or pre-registration. However, as I mentioned earlier, the form of pre-registration can be modified due to specific fields or some personal approaches. You can pre-register as detailed study as you want. For example, uploading all of your research scripts, coding schemes,
08:23
data collection plans, analysis codes and so on. But alternatively, you can also pre-register a more minimal plan. But usually the more detail you provide, a prior, the research by itself, the more confidence readers can have in your results.
08:41
As Professor John Grant wrote, in scientific research like in life, there are costs and benefits to everything. I do not think Open Science practices such as pre-registration does have some labor cost. Of course, it is very time-consuming. Another often-heared critic of pre-registration is that it leaves no room for
09:01
exploration, it is too restrictive. And some are concerned that research RDRs that have been stayed in a pre-registration have a higher risk of being stolen. But as we will discuss now, these fears are misunderstood. Pre-registration actually offer more benefits, which you can see here on a slide.
09:20
First benefit is that pre-registration make your science better by increasing the credibility of your results. Second, it allows you to stake your claim to your ideas later, because it's time-stended, right? Third reason why we should make pre-registration is that front-loading work. It basically means that by documenting what research should do along the way and making it easy to share,
09:44
so regardless of outcomes, the science will benefit through more accurate and complete information. Also for researcher, the work is smooth from after-analysis to before. Also, it can help your future yourself, because sometimes we simply forget what was the original idea.
10:05
This benefit is tied to the fourth benefit, and that's reducing self-deception. Some people cheat, you know, and no one can stop them. But the rest of us should trust each other. However, as Professor John Grown wrote in Science, we can cheat without knowing we are doing it.
10:24
It all comes down to the reliability and probability that a statistic is unlikely to have occurred by chance the p-value. Pre-registration helps remind the researcher of each of these decisions before the data occur. Then the researcher can be clear when
10:42
there are some differences between the exploratory question, which needs correction, to do p-value, and confirmatory research, which does not. And the last benefit, as I already mentioned, pre-registration give us and to our colleague a clear road map to our research idea, hypothesis data collection, and so on.
11:05
You may ask why pre-registration should be part of a tacit writing process. So there are several possible reasons why. Firstly, students will think about their research design right at the beginning of the thesis writing. This will allow them to avoid mistakes in a research approach.
11:23
Also, the workload of their thesis will be more eventually spread over the duration of the studies. By creating a pre-registration, they will learn the principles of transparent and open research. They will also gain knowledge of statistical methods and working with data before they actually get to do analysis itself.
11:46
Students also can increase their reputation and self-image for their supervisors and the thesis opponents. And because students learn about the research process while creating a pre -registration, thinking about their research will help them preview research degree of freedom.
12:02
And of course, in the end, the results of their research will be more robust than it would be without the pre-registration. So how can we incorporate pre-registration in the tacit writing process? On this slide, you can see my suggestion of how pre-registration can be incorporated into the time frame of writing a thesis.
12:26
In a circle, you can always see the name of the time period and above or below the circle, you can see what is the goal of the time period. What is important here is that students get the first feedback on the design of their research in the early parts of their research.
12:47
This early feedback will help students to avoid potential mistakes in their research approaches and also will help them to produce higher quality, more robust tests. In this way, students will not only learn to do a good research practice, but more
13:04
importantly, they will become aware of what their research involves and what they should look out for. Also, in this way, students will receive two feedbacks by the scientific community during the writing process.
13:22
The first feedback will be after they will finish the pre-registration and the second feedback will be during the defense of their thesis. Pre-registration could be also very nicely complemented by a data management plan, if you would like to go even further.
13:44
Data management plan helps students in such a similar way as a pre-registration. It would teach students important information about data management concepts in line with the FAIR principles. At the same time, if students collect unique data, it would be very useful for
14:03
others to make it open afterwards, so other students or researchers could use those data. However, data management is a very time-consuming activity, where I am not so sure if students would be able to do it within the time frame in which they are doing the final paper, the final thesis.
14:26
Of course, it will not be an easy process. This diagram shows the challenges that different stakeholders will face if pre-registration were to become an official part of thesis writing. Firstly, we need to provide the infrastructure to support and encourage pre-registration.
14:45
Of course, it is possible to use off-the-shelf solutions, for example, the Open Science Framework, but it is up to the policymakers and universities to provide the necessary resources for those interested in pre-registration. If pre-registration is going to become an official part of university
15:04
education system, it is necessary to introduce this change into university curriculum. Such a change is unlikely to occur at the national or even global level, I believe. Therefore, I think it is more likely that individual universities will be pioneers and begin to change their curriculum.
15:26
Then there are two challenges that cut across all the stakeholders. It is motivation and communication. Without proper motivation to move towards the trends of transparent and open research, also at the level of thesis writing, the proposed change will not be possible.
15:46
Students, supervisors, universities and policymakers, all of them must be motivated to change the perception of the thesis writing paradigm. From this point of view of policymakers and universities, it is advisable to motivate
16:02
thesis supervisors and students themselves to the principle of transparency and openness in research and to introduce these criteria in the assessment of the thesis, including some appropriate rewards. The next challenge is to provide proper training as well as technical support.
16:22
Here I see a great role for libraries and the use of their technical and knowledgeable base. It is very important that libraries have proper support from universities so that they can provide support to thesis supervisors and to students.
16:41
I believe that librarians together with the thesis supervisor will be the main driving force in promoting the idea of pre-registration of thesis. However, the greatest responsibility will remain to the supervisors and the students themselves. Here I see a great importance of the time allocation for the supervisor of the thesis.
17:04
Thesis supervisors must have sufficient time to devote to the students. Pre-registration is a very time-consuming activity that requires a great deal of time commitment from both the students and the supervisors. Therefore, planning and time management is a key factor here in incorporating pre-registration into the thesis writing process.
17:30
As for the actual process of introducing pre-registration into thesis writing, I believe, as I mentioned, that this process will begin with individual thesis supervisor leading their students through examples of good practice to pre-register their research.
17:46
At the same time, as I already mentioned, I see a very important role for librarians who will become guides for researchers and for students in this area of pre-registration. So, if any thesis supervisor are listening, I would be glad if you consider the possibility of pre-registration for thesis writing.
18:07
Believe me, there are already a number of guidelines you can provide to your students, so it is not so hard. Some of the best guidelines are certainly those by the Centre for Open Science and the Open Science Framework.
18:24
In addition to step-by-step instruction on how to pre-register, these guidelines include also templates for different types of pre-registration. So, if you want to recommend some pre-registration resources for students, I recommend these guidelines. They also have a lot of video tutorials on YouTube, so even those who prefer videos over text, they will learn and enjoy something new.
18:53
If you would like to recommend some more classic resources, I can recommend the book Journey into Open Science and Research
19:01
Transparency in Physiology by Joe Gray, which describes in a very nice way how to make research transparent, including pre-registration. Professor Gray also encourages readers to activity at the end of each chapter with a variety of practical exercises. Also, he provides a number of supplementary materials on the Open Science Framework, so definitely check out this book.
19:30
Thank you for watching this video presentation. I would be happy to be contacted if you have any suggestion for improving my proposal or if you already know some examples of good practice where students are guided to pre-register their research for their thesis writing.
19:48
Thank you very much for listening and have a good day!