Using the Overton policy to academic citation network: how does the policy grey literature and scholarly record connect?
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 30 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/59874 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
6
00:00
Computer animation
00:46
Computer animation
02:06
Computer animation
10:48
Diagram
20:44
Computer animation
23:03
Computer animation
25:03
Computer animation
26:21
Computer animation
Transcript: English(auto-generated)
00:01
Hi everyone, thanks for listening in. My name is Euan Addy and I'm from an organization called Overton.io and today I'm going to talk about how the policy grey literature, the body of policy documents produced by governments worldwide links back to the scholarly record. How does it connect?
00:20
I'm going to speak a little bit about what Overton actually does and is later on but as a kind of background the reason we got interested in this question is because increasingly members of the public especially are interested in what the evidence is behind government policy decisions. So if the government makes a decision that changes your life, what are they basing that decision on?
00:44
What's the evidence? In this presentation we're going to talk about policy documents, scholarly documents, how they all connect like I said. The way we're going to do that is we're going to characterize their relationship using a large set of about six and a half million documents collected from about 180 different countries, from governments in 180 different countries.
01:05
We're going to look at those documents and extract the citations to see what do those policy documents link out to. We'll compare the citation graph to the citation graph you get in traditional bibliometrics. Finally at the end we'll try and answer the question is this data good enough to say anything about
01:23
impact? Can you use it as a university or a think tank to say you know look at my numbers this is a at least a signal that the work we're doing is having an impact on the real world. When Overton talks about policy documents we talk about a wide range of different
01:40
types of report and white paper and legislation and draft bills and all this kind of thing both from government agencies and departments and third sector or even private organizations in some cases so things like industry associations, you know EU agencies, IGOs like the World Health Organization.
02:00
They're all part of the kind of policy making sphere even if they're not government. The reason for that is because if you start off with a simple definition of a policy documents is documents written by policymakers for example you capture things like white papers and draft bills and legislative transcripts, so that's what's being said in Congress or Parliament or
02:25
whatever other kind of legislative body exists in your country which is good, but it's only part of the story and the reason for that is the closer you get to the legislation the less likely it is you'll find any evidence being cited any kind of basic research evidence for that
02:42
you need to go a little bit further downstream. Usually there's a translation step in between any scholarly work and the kind of final legislation and it could be an intermediate stage within government so a government white paper or green paper or report and a technical report from an agency or it could be somebody completely outside of government who fulfills that purpose
03:07
so the Commonwealth Fund does it for health in the US for example Institute for fiscal studies in the UK lots of different think tanks and NGOs charitable organizations again industry bodies Lots of groups do the research on behalf of policymakers, and then they turn it into a message
03:24
That's easier to understand it to act on you know has some clear recommendations and this kind of thing So if we're going to look at how scholarly work Connects to policy work. We're going to need to bear those kind of third sector organizations in mind and bring them into the fold as well
03:41
So for this kind of project often this definition changed from just documents written by policymakers to written by or specifically for policymakers To include those documents that I just mentioned the things like the policy briefs and think tanks You know working papers from central banks research reports from NGOs
04:00
And even to some extent things like clinical guidelines from health agencies And once we have those we can pull them all together into one big network that has all three kinds of document So the the final kind of legislative documents You know the traditional policy documents the scholarly work that they're Basing everything off the basing these decisions off the evidence
04:23
And then the bits in the middle of the reacting as the intermediary as the knowledge broker So I said I'd mention a little bit more about what often actually does and is Open is a large database of these policy documents again in the broad definition of the term It collects about six and a half million different documents
04:42
About 180 different countries so all sorts of different places and languages which we'll cover in a minute and importantly It looks at their citations and outbound links so that could be kind of you know bibliography type citations Or it could be actual web links Actually in reality we get it from everything from you know page footers and table captions
05:05
And this kind of thing is as well as more structured places So we get those citations then we try and link them back to the scholarly record and to people and institutions and things like that Because we've been doing this for about two years now. We've got that database of six and a half million policy documents
05:24
We can already start to answer some some interesting if relatively high-level questions That's a quick aside And this is the the Great Literature Conference, so I'm sure everyone here is very familiar with these challenges in different areas
05:41
but It's quite easy to say okay. We'll just collect everything that's written by a policy maker or for a policy maker of course turning that You know putting that into practice is is much more challenging So there's very few standards or clear definitions of anything. We've just talked about what a policy document is
06:00
Academic publishers have a very real commercial imperative to make outputs discoverable, right? journals want more people to To read their output or to subscribe to their output potentially and certainly to cite their output Governments don't have that incentive. They have a kind of a public mission, but it's not tied necessarily
06:20
To anything that's going to force them to make changes quickly There's no central Identification or you know method of deduplicating policy documents across different sources or anything like that No central search engines. So that's what we're trying to do with with Overton Uncentralized repositories of documents are very rare
06:42
So you can't just go to one site and expect to get all of the documents from a national government Usually you have to go from department to department Agency to agency you have to keep track of which departments have closed down and merged and which agencies No longer exist and all this kind of thing That said some countries do have these centralized repositories. So the UK Ireland and Canada probably
07:03
Have the closest thing to that and in some cases for federal documents, for example in the US They're kind of on their way as well. Even if it's not quite there yet So in general, it's hard to find collect and clean these documents, but it is possible
07:21
this Hopefully isn't intuitive to everyone but of course most policy documents aren't in English Just because most countries don't speak English That said more are in English than you might expect And that's partly because anything with an international audience tends to be at least translated into English
07:41
It's still that you know, the lingua franca so think-tank reports for example In Germany or France if they're kind of aimed at a wider audience than just a domestic one Will often appear as English as well and of course English is still an official government language in some places like India and Singapore and the EU that account for quite a large number of different documents
08:10
We've got because of that centralized repository question. Partly. We've got some gaps in our coverage internationally, so if you look on this map The counts here that it's referred to in the legend that's in the counts of policy documents that we've collected from each country
08:27
So In red in the US is where we've got the most documents from so we've got about 1.8 million different policy documents from there That's partly because we're collecting at state level at the federal level There's a lot of IGOs and think-tanks based in Washington DC in New York. It's just a very kind of policy heavy
08:44
Kind of country that has a you know Kind of long-standing tradition of making government documents available I mentioned Canada and the UK is two countries that have done quite a lot to centralize their documents So again, they're quite well represented
09:01
But there's quite a lot of countries especially if you look at Africa sub-saharan Africa Especially where there's like a band of gray there across the continent. We have very few documents And that's because we're online. We're an online database. We're not dealing with paper and Some of these countries just don't have any policy available online in any kind of way that we could collect
09:26
Sometimes it's because it's not being put online. So the government isn't digital yet and sometimes it's because there is literally no policy So if you think about South Sudan, for example, there's very little kind of written bureaucracy happening there and certainly no kind of written policies being put up on a website somewhere because
09:51
Policy documents come from lots of different countries lots of different types the citation norms and styles vary wildly So this is a big difference between the scholarly world and the policy world
10:05
In the scholarly world if you submit a paper to a journal, I'm sure everyone's been through this and you know God forbid you fail to format the references correctly. It might be sent back to you Okay, there's like there's a reason for you to cite it in MLA or Chicago style or Harvard style or or whatever
10:24
but policymakers have no such need to conform to citations norms and They tend to cite whatever whatever style feels best on the day Without being unkind and you know fair enough. Maybe that's the better system So sometimes the style changes from chapter to chapter
10:44
sometimes from from page to page Use an example policy document this one actually isn't too bad. So this is a Report from the House Natural Resources Committee. So that's a committee in US Congress
11:01
They've produced this report on a hearing about the role of public relations firms and preventing action on climate change and here There's no there's no bibliography section or anything like that, but the they have evidenced things that they're saying with footnotes And when we look at the footnotes they they are somewhere in between Traditional references and something much more informal so we can see on the left
11:25
The first reference there has a DOI link in it, for example, and it's fairly clearly formatted There's a journal reference and there's a book on number three as well And then that's actually it happens in policy documents But it's a bit more rare than what you would see on the right hand side, which is more common
11:43
Where the things being referenced are themselves grey literature, they are themselves policy documents and this happens a lot where Sometimes hard to parse out exactly what these things are without kind of knowledge of the context of the document
12:03
If we look at policy documents from different sources And we kind of break it up into how many of them are citing scholarly work So things like that that journal article and book and how many are citing other Policy sources, so you know other reports from the policy world other bits of legislation that kind of thing
12:22
The numbers vary widely across sources so GovUK I mentioned being a centralized repository again in the UK for for lots of different departmental outputs And of them about 8% of them cite at least one scholarly paper So 8% of all of the documents on GovUK and about 5% cite other policy sources
12:43
And that's because it is a very generalized repository Of everything the government's producing if you look at something a bit more focused like the Congressional Research Service So the Congressional Research Service their job is to go out and do kind of reviews of the literature or the current kind of policy landscape and then to
13:01
Synthesize that into a short document that's then distributed to Congress people to read and to inform them 23% of their output cite a scholarly paper So they're relying much more heavily on this the scholarly world for evidence and 33% of the documents cite other Policy sources so often that's going to be think tanks
13:21
lobbyist reports NGOs Charitable foundations all that kind of thing In general in terms of how much of the scholarly literature is cited It's a bit like traditional bibliometrics And there's a little bit of a long tail
13:41
So a lot of papers or there's a set of papers rather they get cited very frequently And there's a long tail of papers that get cited kind of once or twice in general a few people have a few groups have looked at this so one in CWTS And Leiden and another one science metrics in Canada, which is now owned by
14:03
Elsevier or size metrics is owned by elsewhere say They both like this they've found similar kind of numbers, which is around between kind of five and eight percent Of articles End up being cited in policy at least once So the the bullet point I've got here on this slide is from the science metric study
14:24
And they took all of the Scopus articles all the articles in Scopus the abstract and indexing database published between 2008 and 2016 So they've had long enough time to be cited in policy And then they checked them in overton to see you know do they have at least one policy citation I came back with about yeah, well five point eight percent of them
14:45
Do you have at least one citation and policy Interestingly it varied a lot based on the subject so To test this they took random samples of articles in different subject areas and Of the development studies sample for example they found forty seven percent were cited at least once in policy, but of the drama and theater
15:08
sample only zero point one percent Were cited in policy so it kind of makes sense intuitively But the data bears it out
15:21
Interestingly when you look at all those subjects together You see something markedly different to what you would expect if you were coming from the scholarly bibliometrics world So in scholarly bibliometrics, it's the hard sciences that get cited the most and the social sciences are very much the the poor Cousin so you know high energy physics and cancer get a lot of citations and
15:45
social sciences don't In the policy world it's the opposite well not quite the opposite, but it's different It's flipped the social sciences and humanities is the kind of broad subject area they get cited the most in Policy and actually the harder the science you are if you're you know organic chemistry or quantum physics or something
16:05
The less likely it is you're going to be cited at all This makes sense Policymakers just aren't citing quantum physics papers Well, they are citing if you look on the right hand side the right hand graph We've broken up social sciences into the different kind of sub disciplines is
16:21
economics and psychology And That said some parts of the the life science especially are still cited a lot if you think about public health for example or during the COVID-19 pandemic obviously lots of lawmakers and policymakers
16:41
Were busy citing med archive and other medical journals, so it does absolutely still happen But it's the social sciences that tend to come out on top Policy documents also tend to cite locally So this isn't necessarily a surprise when you think about When a policymaker goes looking for evidence they're going to look for
17:03
Local expertise you can you know come in and be embedded with the department or can come and give evidence to committee Or they'll pull out a call for evidence and the people kind of most Rewarded for responding to those calls for evidence are the people in the country it came from But if you look at all the policy documents from Australia
17:21
And then all of the scholarly research that they cite as evidence And then you look at the affiliations of those scholarly articles The top three affiliations the affiliations that crop up most often are all Australian, so you know who's giving Who's providing most of the evidence? Or evidence most frequently rather to Australian policy documents. It's Melbourne Sydney and the ANU
17:46
Equally in the US. It's three US universities Harvard Stanford and Washington in Sweden Three Swedish universities and Brazil is three Brazilian universities and that holds pretty true for most countries The exceptions are smaller countries where I think they maybe don't necessarily have the research
18:06
Base to be able to answer all the questions required by policy makers so Singapore for example. They're the top three universities In terms of the numbers of papers being cited by government there Aren't Singapore in there from elsewhere in the world
18:26
When you look at policy to policy citations so this is a government policy document that cites one of those knowledge brokers a think tank for example or a Think tank citing another think tank You get another surprise
18:42
Compared to what you might expect if you're coming from the scholarly world Which is that there are a lot of citation networks or Actually, what what would tends to be classes of citation cartel in the scholarly world? This would get you a kind of slap on the wrist and get your journal delisted from from Web of Science for example
19:01
But between think tanks, there's there's no such Problem, so it's not considered a bad thing to have a citation cartel in think tank band. It's considered good marketing, right? So what you're doing is you're amplifying the voices of the people who agree with you and who are Promoting a similar message to you and there's there's nothing wrong with that. It's just a different way of approaching it
19:23
But it does mean that when you're doing any citation analysis you have to be aware of this kind of thing like it very much exists and it's encouraged rather than punished in terms of How fast is policy of picking up research?
19:43
We were interested in this question because again, I mentioned covert 19 earlier We were seeing articles being published in med archive and the preprint server And then appearing in policy, you know two weeks later. So the turnaround time was really really short a Lot of policy then was being written by experts kind of drafted in or being repurposed from from body site the WHO
20:08
It's partly accounts for that. Um More generally though when we look at all the articles for example published in 2010 And then we graph, you know, what was the year where they were first cited the kind of half-life of that graph is about
20:24
Just under four years after publication So what we're saying is in general on average, so obviously it varies a lot based on discipline, especially But on average it takes about four years For a piece of scholarly work to start appearing in policy at the beginning
20:48
I mentioned that we were going to tie all this back to to impact and ask the question, you know Given that we can extract these citations between policy documents and between policy and scholarly documents Is it something we can use to measure?
21:02
impact So Practically speaking. What we're really asking is can I as a university? Can we look at this policy data and say, you know We're doing better or worse than expected or from compared to our peers in Influencing policy either locally or internationally or as a think tank. For example, can we say that we're doing a good job of
21:27
Influencing government policy because our work is being cited more often than our peers or than you might expect random and I mean the spoiler is tentatively the answer is is yes I think there is enough of a link there and I'll talk about that in the next slide
21:42
But one important thing to remember is that a citation itself is not impact. It's just a pointer it's an indicator that impact did happen and the reason for that is because after a policy is Made is put into legislation has made law if it is if it does get to that point
22:00
That doesn't necessarily mean it changes people's lives. So If step one is being cited there is a step two Where then gets put into practice and it changes things and you know, that's the the best outcome Hopefully it changes lives for the better and not for the worse. That's a different kind of impact But there are other possibilities as well. There's a step two in policy, which isn't you know, infrequent
22:21
It's not uncommon where the policy is implemented, but it's then kind of forgotten quietly forgotten about, you know It doesn't really make any difference in when it's technically true But in practice not makes no difference or another step two where it's implemented And then you know the incoming administration two years later comes in or even earlier comes in and immediately scraps it
22:43
So perhaps even before I had time to be picked up and properly acted on so Can we say for sure the impact happened if you're cited by policy? No, you can't because we don't have all of the available data But what it does do is highlight that that pathway happened that there's a possibility of it
23:05
so all that said and that was quite a Like a drastic caveat But all that said we we do see correlations between real-world impact and policy citations and one example of that Was a study
23:22
We published recently or at least we published the beginning of it in QSS the journal and We looked at data from ref in the UK Which I won't go into a lot of detail about the ref here But basically it's a mechanism by which every four or five years the UK government
23:43
Assesses the work of every university in the UK and it determines how much block funding they get from the government And the way it works is universities write case studies for the impact of their best work and all those case studies are peer-reviewed centrally and scored and then you're the scored kind of the
24:04
Complicated weighting system, but the higher the score the more funding you get from the government as a university So it costs a lot of money to run a lot of people spent a lot of time writing case studies It's a very big deal for universities And they release the data afterwards somewhat redacted. So it makes it hard to
24:21
Do analysis and detail on it. But what we can see is that there's a correlation between the number of highly cited in Policy articles that the university has so the more articles that a university has produced that are cited a lot in policy and The number of highly scoring case studies to do with policy impact it has
24:44
So that's the human written case studies, but you know, we did this work and influence policy And then another human has read it and said yes, that does sound like a real impact to me You know five out of five or or less as the case may be so that correlation does exist So that implies that there is something there
25:04
Okay, thanks very much for listening. I'm gonna wrap up quickly now just in summary So first of all when you're looking at government documents when you're doing any study of government documents, especially linking it back to to other Groups like scholarly world and you probably also want to consider these policy adjacent third sector organizations include them too
25:22
They're an important part of the ecosystem policy documents Not surprising again to anyone here in the conference But they come in a range of different styles and sizes. It's very difficult to clean and collect and normalize them They lack many of the standards we take for granted in the scholarly world. Certainly They are connected in a very significant way to scholars and scholarship
25:44
They cite a lot of research and they cite it frequently Both research from universities and other kind of grey literature research if you like So they cite a lot of research from IGOs and think tanks and private organizations In those citations we see more social sciences links and kind of more hard science research
26:03
And it does seem to be that link between policy citations and policy impact assessed by human beings Thanks very much. I'm happy to take questions later or you can Find my twitter handle and email in the speaker bios. Thank you