65. Helmholtz Open Science Online Seminar
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 5 | |
| Author | 0000-0001-8249-1752 (ORCID) | |
| Contributors | 0000-0002-7060-3670 (ORCID) 0000-0001-6967-9443 (ORCID) 0000-0001-7080-634X (ORCID) | |
| License | CC Attribution 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/59552 (DOI) | |
| Publisher | 0281dp749 (ROR) | |
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Computer animation
04:19
Computer animation
04:48
Computer animation
05:25
Computer animation
11:50
Computer animation
18:12
Computer animation
18:47
Computer animation
24:59
Computer animation
31:12
Computer animation
34:56
Computer animation
Transcript: English(auto-generated)
00:00
Welcome to today's online seminar on Open Research Information for Responsible Research Assessment. Thank you very much for joining and this is our 65th online seminar of the Helmholtz Open Science Office and we're very happy to welcome you here today. As I said, today's topic is Open Research Information for Responsible Research Assessment
00:21
and our speaker, Professor Dr. Ludo Waldman, who will introduce you in a few moments, will explain why openness of research information is a crucial prerequisite for more responsible approaches to research assessment. He will discuss various initiatives promoting openness of research information both in the Netherlands and also globally with a
00:43
specific focus on metadata of scientific publications and he'll also discuss how organizations can support these initiatives and participate in them. So we're very excited to be hearing from him and discussing with you. My name is Lea Ferguson and together with my colleagues Nina Weissweiler and Antonia Schrade, I very much look forward to today's talk and
01:04
the discussion with all of you afterwards. And to get us started and to get to know us a little bit, we prepared a few questions that we'd like to ask you and we'll start with the first one now. So we're interested in what main field of research or what main field in academia you're
01:23
currently active within just to get a feel for our group. So feel free to answer in the poll or if it's another option just write in the chat. And while you choose your answer, just a few organizational remarks and please mute your mic during the talk so we can listen to Ludo
01:41
and please note that the session will be recorded. You got the reminder at the beginning of the meeting but for the ensuing discussion time slot we will stop the recording so no worries there and just the main talk will be recorded. The slides and the recording will be made available to you after the online seminar via our event website and your questions throughout the
02:01
talk can be posed in the chat. Please either write your full question there or indicate with a question mark that you would like to ask a question and then when prompted you can unmute your mic and we can get into the discussion. And we'll have an open discussion time slot after the presentation by Ludo so feel free to write in the chat as we go along so we can collect the
02:21
questions there. Okay I think we can then finish with the first question. So as you see quite a few have joined us from the library context, quite a few from administration and also research and some other research areas and I think this is going to be very interesting
02:41
especially when we talk about the metadata of scientific publications but also the tools and yeah organizational aspects that we can look into into this field. So yeah thank you very much for sharing your replies and now I'll hand it over to Nina. Yeah very warm welcome
03:01
from me as well. Thank you all for joining us today and thank you to Ludo for being here. I would like to introduce him to you. He's our speaker today and he's a professor of quantitative science studies and deputy director at the center for science and technology studies at Leiden
03:22
University in the Netherlands. He's also associate director of the research on research institute and his work focuses on developing new infrastructures algorithms and tools to support research assessment science policy and scholarly communication which are all topics we want to discuss today. Together with his colleagues Nees Jan van Eyck, Ludo has developed the well-known
03:45
BOS viewer. It's a software tool for biometric visualization and he's the coordinator of the CWTS Leiden ranking which is a bibliometric ranking of major universities worldwide.
04:00
He also coordinates the initiative for open abstracts and in addition Ludo serves as the editor-in-chief of the journal quantitative science studies. So we are really excited to have you here today. Thanks so much for joining us Ludo and the stage is yours and we
04:22
would like to express my gratitude to the organizers for setting up this seminar. It's great to be here, a topic that I care a lot about and I'm sure you do as well, so making research assessment more responsible and the role of open research information in that endeavor.
04:44
So what I will do in the next half an hour or so, I will say a few things about the developments around responsible research assessment without going into a lot of details because I would like to then zoom in on the connection with open research information.
05:03
I would like to say a few things about developments around infrastructures and openness of all kinds of data and metadata. I would like to say a few things also about developments in my own country the Netherlands which might be of interest for perhaps Germany as well
05:21
and then I will end by drawing a few conclusions. So what we see here is some developments in responsible research assessment. So basically a movement that is trying to make practices for assessing researchers and research units more
05:43
responsible and an important element in that development is trying to prevent an over-reliance on bibliometric indicators. So what you see at the left is the declaration research assessment and if you may be familiar with it, it was launched almost 10 years ago.
06:02
It's a statement that is in particular very critical about journal impact factors. It's a statement that is also supported by many organizations and individuals who make a commitment to reduce their reliance on impact factors in the context of research assessment. At the right you see the Leiter Manifesto which is a statement that was
06:25
published by a number of colleagues and also by myself about seven years ago. It's a statement that presents 10 principles for responsible use of research metrics. On one hand this statement emphasizes the value that we believe research metrics have. So for instance bibliometric
06:46
indicators, indicators based on publication citation statistics. It's a statement that emphasizes that there is value in these types of metrics but at the same time it's also a statement that urges evaluators not to overuse and not to rely too much and too strongly on
07:09
these types of indicators. And that is what we consider to be responsible use. It's about finding the right balance between research metrics on the one hand and
07:23
expert-based evaluation on the other hand. So there are many more developments in this area. In my country in Netherlands we have a very important and I must say also very inspiring initiative. It's called Room for Everyone's Talent. It's about changing the ways in which
07:41
academics are recognized and rewarded. And again this is partly about moving away from an over-reliance on impact factors, years indices and other types of bibliometric indicators. It is also about broadening the perspective of how recognition and reward is organized. So making sure that also for instance contributions to teaching get properly recognized.
08:06
It is also about making sure that other types of research outputs and research activities different from traditional outputs like journal articles that these other types of outputs and activities also get properly recognized and rewarded. This might also include activities in
08:25
the area of open science or activities related to promoting a healthier research culture. All these things are being taken into consideration in this initiative that we have in the Netherlands at the moment. And this is an initiative I need to emphasize that is supported by all the
08:42
Dutch universities and also the other major research organizations in my country. So this is an important development in the Netherlands. What we now see at the European level is a kind of a similar development. Some of you may have seen this and may be following these developments. So quite recently the Coalition for Advancing Research Assessment
09:04
was launched. It's an initiative by Science Europe and by the European University Association. The European Commission also plays an important role. And this coalition consists of organizations that have signed an agreement in which they make a number of commitments to
09:23
reforming research assessment and moving towards more responsible ways of doing research assessment. And to a significant extent this is aligned with some of these earlier initiatives that I showed on the previous slide like the Leiden Manifesto. Then of course there are all these developments around open science and in particular I want to
09:45
draw attention to the UNESCO recommendation on open science which I think is interesting. First of all because it offers a very broad and rich perspective on what open science is about, going much beyond the things that we traditionally discuss under the heading of
10:02
open science, open access and open data. This is a much broader and much more inclusive perspective on open science. This recommendation is also interesting because of the global support it got from basically all the UNESCO member states.
10:21
So there are many, many things in the UNESCO recommendation on open science. I would like to briefly highlight a few things. So there is this idea that there's a need to review research assessment practices, the way the scientific career system works and to make sure they are aligned with open science principles. So basically this is about making sure that
10:44
people get recognition for efforts related to open science that they make. So of course people often make the argument that for instance making data open available takes time and people feel that they need to be, they should be recognized for making these efforts. But the other way around, the recommendation also
11:07
emphasizes the importance of broadening the way in which research evaluation is done and making sure that we recognize the diversity of outputs, activities and missions. And again that's
11:23
very much in line with these developments that I showed on the previous slides. So what I would like to emphasize is that I think there is a really important connection between these developments. On the one hand this movement that is promoting
11:41
reform and research assessment and on the other hand the open science movement and the two are reinforcing each other in a mutual way and I will explain what I mean by that. So first of all in order to facilitate and enable responsible research assessment
12:03
I believe that it is crucial that we have research analytics that meet certain preconditions and in practice actually at the moment they often do not meet these conditions. So responsible research assessment is research assessment where decision making, where evaluation is primarily done by experts based on their expert knowledge. But these
12:25
experts are supported by indicators or metrics if you like the term, by data, by perhaps all kinds of research analytics. These research analytics need to be transparent. This is important
12:40
because that means that experts that use these analytics can reflect on the meaning of these analytics by actually also questioning them, by trying to understand what is behind for instance a certain number, trying to see what is the data behind that number, how was this data transformed to get that particular number that in the end is provided to the evaluator. So rather
13:06
than just blindly relying on certain numbers like an impact factor or an hair syntax or something else, evaluators should be able to also question these numbers and to debate what these numbers actually mean and that requires a certain level of transparency. We also need pluralism,
13:25
pluralism in the sense of enabling different perspectives on what we can learn from particular data about what's happening in the research system. So rather than having one particular interpretation, for instance an interpretation that is provided by one specific indicator,
13:47
we should enable anyone who is involved in research assessment to develop their own perspective on what the data actually tells us and how the data could be interpreted. So we should provide room for pluralism, for a plurality of different views on what the data tells us and then of
14:05
course we can have a debate, we can have a debate about the merits of the different interpretations of the data and ultimately I think this leads to something that I call a more democratic way of working with research analytics. So at the moment what we often have
14:22
is we work with computer systems, we subscribe to particular computer systems, they give us particular metrics. These metrics typically are not very transparent, usually there's some explanation of how they are calculated for instance but you cannot typically see the underlying data and re-analyse that underlying data. You cannot transform that data into
14:44
alternative types of analytics, so you don't have that level of pluralism that is needed and ultimately that means from my point of view that a democratic discussion about what the data does and does not tell us is hard, it's hard to organize because of the constraints that
15:02
the tools that we typically work with, the research analytics tools impose. So I feel we need this transparency, this pluralism and in the end we need democratic research analytics and this is possible only if research information is open and that's where we have the connection
15:21
between responsible research assessment and open science. So essentially it's a two-way connection. On the one hand, and this is the argument that is often made and it was also for instance visible in the UNESCO recommendation, if we want to promote open science we need to make sure that we give recognition to open science practices. So people that indeed practice open
15:45
science should be recognized and rewarded for doing that. This I believe is important but I'm at least as interested in the opposite thing from open science to responsible research assessment. I would argue that if we want to make research assessment more responsible,
16:04
this can only be done, as I showed on the previous slide, if assessment is done based on openly available research information so that we can have a genuine discussion about what the information in the underlying data, what it tells us and what the implication should be for the assessment. So the research information should be open, so essentially the research information
16:25
should be compiled in a way that aligns with open science principles. So if we want to have both, both arrows, then on the one hand we have responsible research assessment which of course takes into account a broader diversity, a broader range of activities and outputs.
16:44
We have this responsible research assessment as a way to also give recognition to open science practices and thereby to strengthen the movement towards open science. But at the same time,
17:00
in your direction, open science is strengthening the responsible research assessment movement by pushing for openness of research information, which I believe is an essential prerequisite for responsible assessment practices. So this is for me the picture within which we are operating at the moment. And what is kind of a challenge is that we need to do two things at the same time.
17:25
We need to reform the way we do assessment, we need to move towards a more open way of practicing science, but this needs to be aligned in a very careful way. This challenge is also recognized for instance in UNESCO recommendation, which is mentioning imports of open
17:42
infrastructures including in their terminology open bibliometrics and scientific systems for assessing and analyzing scientific domains. So that's essentially the same point, at least in my interpretation, the same point that I was making on the previous slide. What I will now show you is where we are on this journey, on this journey towards more open research
18:04
information, where are we, and also where do we still have major challenges, major problems. So I'm going to show you from an infrastructure point of view the current situation. This is not comprehensive. This is a little bit kind of, in some sense, perhaps still traditional because
18:23
this is mainly about traditional research outputs. And I want to emphasize that this is definitely, this should not be the only thing that matters in research assessment, but it is of course something that we're all quite familiar with. And I will actually show you that even for these relatively traditional research outputs, we still have major challenges in terms of
18:44
open research information. And the challenges are perhaps even bigger for other types of outputs and activities that should be recognized and rewarded. So some of you may know that there has been the initiative for open citations, which was launched four years ago, and which is promoting openness of citation data. So citation data of course plays an important
19:07
role in research assessment. Perhaps sometimes it has too much weight in research assessment, but I do believe that it has a lot of value and it makes sense to take this information into account. But the information should be open and the initiative for open citations has been
19:21
pushing for this. And what we see now is that this initiative was launched five years ago in 2017. So one year ago, this blog post was published, which made the argument, the point that actually the open citation data now approaches parity with what is available in proprietary commercial data sources like Web of Science and Scopus. And more recently,
19:44
actually just one month ago, there has been another milestone and now all citation data that scholarly publishers deposit in Croslev, you may know that's an infrastructure that many publishers work with, all that citation data is now made openly available. So in the past,
20:04
it was made openly available only if publishers agreed to open their citation data. And many publishers initially did not agree with that, but now all the citation data is openly available. So Croslev has been able to change its policies. And this is at the bottom
20:22
of the slide, you see a recent news piece in nature that is kind of announcing this development. So essentially, we can now say that most citation data is openly available. Not all citation data, there are still all kinds of gaps in what is openly available. It has to
20:41
do with, for instance, limitations of the Croslev infrastructure. I won't go into the details, but essentially we have gone through this development where much more citation data has been made openly available. And that means we can now all, everyone can construct citation indicators. So you don't need to rely anymore on research analytics that you
21:05
obtain from commercial software tools. And you may use those and perhaps it's fine to use them for certain purposes. But if you want to, if you don't trust these solutions, if you don't like them, if you disagree with the metrics they provide, you have the opportunity to create your
21:20
own citation metrics because all the citation data is openly available. The initiative for open abstracts is a similar initiative that is pushing for openness of the abstracts of scientific articles. It's requesting publishers to make these abstracts openly available in Croslev. So here you see that many publishers have indeed taken that step. They were shown in orange in
21:42
this graph and on the horizontal axis you can see the share of all their abstracts that are openly available. Unfortunately, most of the largest publishers have not yet taken that step. So they are shown in blue and they still need to take this step. They still need to make their abstracts openly available. At the moment, they have not yet made that decision, unfortunately.
22:05
Why is this important? This is, for instance, important because research analytics around, for instance, the sustainable development goals, they typically rely on text mining of scientific articles, in particular the abstracts of scientific articles,
22:21
to determine which articles relate to particular sustainable development goals. So at the moment we have commercial providers that use proprietary data to do all this text mining for us. They make particular choices in the way they do it. For instance, in the way we disagree, if we disagree, we have no way of doing this ourselves, in a different way that
22:45
perhaps is more in line with our particular viewpoint on sustainable development goals. Openness of abstracts will enable anyone to do this in the way we like most and the way that fits best with our purpose, the purpose for which we are interested in a sustainable
23:02
development goal. So that's why this is important. I do want to mention that next week is open access week and then there will be some announcements around next steps in this initiative of open abstracts. So together with my colleague, I recently did a systematic analysis
23:20
of the degree to which this type of metadata, satacious, but also other types of metadata, is openly available in Crossref. So you see in this graph that over time more and more of this data is made openly available. So that's also thanks to publishers that are doing an increasingly good job in making metadata of their articles available in Crossref. But still unfortunately
23:44
there are quite a lot of publishers that actually fail to do this in a proper way. So here you see for instance affiliation data. This is data that is for instance crucial to create statistics, research analytics for institutions. You need to know which outputs belong to which
24:02
institution and that means that publishers need to make affiliation data, the affiliation data that you find in scientific articles, they need to make it openly available in Crossref. So some publishers are indeed doing this. So they are shown at the right of this graph like
24:20
Wiley and Taylor and Francis, some of the bigger ones, they do a good job. They make this data openly available. Other publishers elsewhere, Spring and Nature, MDPI for instance, but also many others, they do not do this. So this data is made available in proprietary sources like Web of Science and Scopus. It's not made openly available through Crossref. And for instance
24:44
university rankings, these types of tools, they cannot at the moment be created in an open, transparent way based on open data simply because a number of big publishers don't make the data openly available and only proprietary data sources can be used for this. And that's also one of the
25:02
reasons why university rankings suffer from a lack of transparency. There are more developments, so there's also the importance of having persistent identifiers for organizations and you may know that the research organization registry is working on this. There's also open funding data, yet another important thing. Here we see that publishers are doing a somewhat better job. So we see that
25:24
many publishers, also the big ones, are making an effort to make funding information openly available, but often there are gaps in the data they provide, unfortunately, and only for some of their outputs funding data is openly available. And for many outputs we don't actually know whether they were funded by a particular funder. This for instance is important to
25:45
check compliance with mandates of funders, like mandates for open access publishing. It's hard to check compliance if this data is incomplete. So recently together with a colleague at the European Research Council, I looked at the extent to which funding data is
26:06
made openly available for research funded by the European Commission and we also made comparisons with the data that's available in proprietary data sources. And what you see is that there is a lot of openly available funding information, but there are also major gaps compared to the
26:25
information that you can find in Web of Science and Scopus. And actually this is due to particular publishers that failed to make this data openly available. So at the right, this chart, this bar chart shows which publishers are doing a good job in making the data openly available, but also
26:42
which publishers are not actually making the data openly available, while they do at the same time provide the data to, for instance, Scopus and Web of Science. There's more, there's many more types of questions around metadata that is really important to have. So for instance there was this
27:02
recent blog post which was emphasizing the importance of links between preprints and journal articles. So for instance when a journal article is retracted, it's important to know that also the preprint might not be reliable, but in order to have that knowledge you need to know that the preprint and the journal article are just two different versions of the same article, so you
27:23
need to have a connection between them and these connections are often missing. So that's yet another important piece of metadata that we need to have. So Crosstalk provides these dashboards where you can see how different publishers are performing in terms of providing metadata.
27:41
So at the left you see Royal Society, they do an excellent job in making metadata openly available. At the right you see the American Psychological Association and they have a very poor record in terms of openness of metadata. Even the citation data is not open which is quite disappointing I must say. This is of course also something institutions can use when
28:03
they negotiate new agreements with publishers. This is something institutions can use to actually also make sure that publishers do not just provide access to their content and possibility to do open access publishing, but also provide open as well as metadata which ultimately will
28:23
enable research analytics to be made open and transparent and this will contribute as I mentioned to more responsible research assessment. So essentially the argument that I'm sometimes making is that we need to shift the discussion a little bit from open access to
28:43
fairness of publications. Publications being fair in the sense of findable, accessible, interoperable and reusable. Of course we know the whole debate about fairness of data, but publications just like data I believe should be fair. So if we ask for publication to be open access we are asking for accessibility and reusability, but I would argue that findability
29:03
and interoperability are just as important and that requires openness of all this metadata because that will enable discoverability or findability of the publications. It will support responsible research assessment practices and that's why this is really
29:22
of key importance. Recently the same argument was also made in this editorial in Nature, it's quite interesting. So Nature is of course published by Springer Nature. Springer Nature has kind of a mixed track record in terms of how they make metadata openly available.
29:41
So they make citations openly available, but they don't make all the abstracts openly available for instance. They don't make author affiliations openly available, but Nature itself, the journal, published a really strong statement where they urged publishers, so also their own publisher actually, to make this metadata openly available. So they praised initiatives like the open abstracts initiative and they said this is what we need and we need
30:04
all this metadata to be openly available and publishers, including our own, should start doing this. Well there's much more going on, I won't go into the details, but there are all discussions about pit graphs and openness profiles, also really very important in
30:20
developments. You may know this report from quite an interesting project. I won't discuss that in more detail. I do want to say a few things about something in the Netherlands. In the Netherlands right now we have another discussion going on. So we have now seen all these developments around open metadata, open infrastructures, and we see that at the moment
30:46
we are in a movement gradually towards more and more openness, but there's still a lot of steps that need to be taken to push for that and to make sure that we kind of in a systematic way promote more and more of this openness of research information.
31:01
In the Netherlands we now have seven guiding principles for open research information, principles that are supported by many of the major research organizations in the Netherlands, and I want to share that with you because this might also perhaps be something that could be of use in, for instance, the German context. So a number of experts in the Netherlands got
31:21
together and they proposed these principles and these principles were then adopted by the major research organizations. Let me just very quickly go through the principles. So the first principle is trusted and transparent provenance. So whenever we have
31:43
particular data or metadata, it needs to be clear where it comes from, what is the source, the origin of the data. The second principle is openness of metadata. So that is very closely linked to the things I just discussed. So we need the data to be open. And as I mentioned, that is essential for responsible assessment. The third principle emphasizes the importance
32:04
of also the algorithms being open. So just the data being open is not enough. When data is transformed algorithmically, for instance, into metrics, we also need to know what these algorithms are doing. So that's just as important as openness of the data.
32:23
Then we have enduring access and availability. So this openness should not be a temporary thing, but it should be something that is a long term commitment to openness of the data.
32:41
So we need to be able to have guarantees that things will remain open also in the future. And then we have open standards and interoperability. So if you just make things openly available, then still it might be of limited use. So we need to do this in a way that is interoperable. So that's also why I was putting so much emphasis on Roslev and also why I mentioned persistent identifiers. That's a way to make sure that the data is
33:10
made available in a standardized way, which makes it much easier for us to work with the data. Open collaboration with third parties. So this is interesting. In this principle,
33:21
it is stated that knowledge institutions must engage in open collaboration where innovation, competition, and public value are recognized and respected cornerstones. What this actually means is that in all kinds of collaborations with, for instance, infrastructure providers, the collaboration should be done in an open way.
33:46
What this also means is that there is room for collaboration with a variety of different types of organizations that we can work with. That also includes commercial organizations, for instance, provided that these organizations are willing to commit themselves to such an open
34:02
way of collaborating. And the final principle is about academic sovereignty. Which emphasizes the need of a governance structure that ensures that organizations in the Netherlands remain committed to all these principles. So this is where in the Netherlands
34:23
we are at the moment in our thinking about how at a high level we can make sure that we make the right decisions in promoting this type of openness, promoting openness of research information. And ultimately these high level principles and, for instance, these governance
34:41
structures should enable the Dutch research organizations to, in a systematic way, push for this openness of research information. So that's what I wanted to share with you about the situation in the Netherlands. There are many other developments like, for instance, the plans that we have for an open knowledge space, but I won't go into the details about that. I just want to conclude my talk by summarizing some
35:04
recommendations. So what I hope I have made clear is that in order to move towards responsible, more responsible research assessment practices, it is crucial that we work with open research information. So that provides transparency, pluralism and ultimately democracy, a democratic
35:23
way of working with research information and using that information in research assessments. This means that we need to take advantage of infrastructures for open research information, and we need to actively support the further development of these infrastructures. So that can sometimes be about contributing to, for instance, financial sustainability of these
35:42
infrastructures. It can also be, for instance, about institutions in their negotiations with, for instance, publishers making it mandatory for publishers to contribute to these infrastructures, for instance, by making relevant metadata openly available.
36:01
Finally, these guiding principles that I showed you, that is something that we have found in the Netherlands to be a really powerful way to make sure that we all stay on track. It's, of course, always kind of attractive sometimes to just in a pragmatic way choose a certain
36:22
solution that seems to be the right one in the perhaps in the short term, but in the longer term that might actually not be the right way of doing things. It might need to sort of lock in that ultimately is going to provide obstacles to having more open research information and having more responsible research assessment practices. So these guiding principles are kind
36:43
of a way to prevent ourselves from making the wrong decisions and creating situations where we are not moving in the right direction. So I would like to conclude here and I would like to thank you all for your attention.
Recommendations
Series of 5 media
Series of 2 media