All right, so we've got another talk scheduled, but I haven't got those people available yet, so we might skip ahead to Amanda Lawrence. Amanda's a researcher and librarian who works in the areas of research communications,

media, and publishing systems, digital collecting, open scholarship, grey literature, which is what she's gonna be talking about today, knowledge infrastructure, and public policy. She's currently completing a PhD on research communications and public policy in the media communications department at RMIT University. I hope that wasn't out of date. I got it from your orchid, I think.

Amanda was the director of analysis and policy of the analysis and policy observatory, an award-winning public policy research database from 2006 to 2018, where she's managed a number of research, Australian Research Council linkage and infrastructure grants. Amanda's worked in various roles in the book and cultural sector, including establishing the literature residency programme

and the author touring programme at the AsiaLink Centre at the University of Melbourne. So I'll hand you over to Amanda, who's going to give us a presentation on the challenge of institution published grey literature.

Great, thanks, Alex. Maybe make the, I think I'm quite large, so if you want to make the slides a little bit bigger. Yeah, okay, yeah, great. Get rid of me entirely, fantastic. So yes, thanks for that introduction. I didn't realise you were going to,

I'm not sure where that bio came from. It is still accurate, although submission is hopefully going to occur very soon for my PhD and then this week, I'm hoping. So, yeah, thanks everybody for joining in this session.

I'd just like to say first up that I'm not really much of a Wikidata, WikiCite or even Wikipedia expert. I have, I'm a total dabbler, so I can't really, so Margaret's presentation was pretty, a bit advanced for me,

but I do know a fair bit about grey literature, so I'm going to focus on that and focus on why it's really important and why we need to think about how we can get it into Wikidata and WikiCite.

So firstly, so just talk for 15 minutes, a little bit of an overview of what we're talking about here and then some of the kinds of publications we might be wanting to deal with and what the sort of referencing issues there might be in the data model

and some of the challenges that are involved. So firstly, I think Daniel already mentioned the sustainable development goals. So the reason why I want to bring this in is to have a think about just the scale, the complexity, the diversity of topics,

of institutions, of people, of countries, of types of research or research methods that would be involved in trying to address these goals. It is not going to be done

simply by academic journal articles, although they are going to be important. It's also going to involve electoral data, it's going to involve economic data, it's going to involve government research, it's going to involve thousands of NGOs

and think tanks and consultants, et cetera. So if we want to be a source of information and for the sustainable development goals, we need to expand our idea of where research is located. So I came across recently this term requisite variety

and I really like it. It comes from a sort of computing basis. So some of the audience may be familiar with it. It was new to me and it's also being adopted by people looking at sort of evidence-based policy

and we might think about that. So the quote is, an effective evidence ecosystem would seem to need a requisite variety of initiatives to encourage and enable evidence use. That is the repertoire of initiatives to promote evidence use needs to be at least as varied and nuanced as the policy and practise context

that are being targeted. So the point is that the kinds of resources that we're going to need are going to have to be probably as complex and diverse as the problems that we face. And therefore the computing system that we provide is also going to need to be fairly complex.

So the kinds of reports that we might be thinking about in this context of the sustainable development goals and climate change, et cetera, include reports from the IPCC itself, clearly peer reviewed. So it's not true that research reports

from organisations aren't peer reviewed. UN, the environment programme, biodiversity issues and World Bank, major quasi-government organisations and international non-government organisations.

There's also a huge variety of reports produced at a national, at a local level, all around the world. So these examples are all from Australia, but they include research centres, often research centres in collaboration

with both companies and NGOs, advocacy groups, government research centres, et cetera. So my concept of what we're dealing with here is that if we think about there's research and there's research communication

and scholarly communication is a subset of that, but so is research publishing and they kind of overlap. There's commercial academic publishing and there's organisation research publishing. So a lot of what we have on Wikisite and Wikidata is commercial academic publishing. There's also market research publishing, where you can pay thousands of dollars

to get access to a market research report. And there's substantial, but still relatively small, non-profit academic publishing. So the open access movement, diamond open access journals, et cetera. So if we think about what might be then included

on Wikidata and Wikisite, we need to think beyond books and journals to think about reports, technical reports, evaluations. It's not quite clear what that type is referring to.

There's policy briefs and briefings. Again, not very well-defined on Wikidata. Discussion papers, it's on Wikidata, but actually to my mind is the wrong definition. White papers produced by consultants and governments as well. Systematic reviews, again, on Wikidata,

that's kind of looks a bit like a topic. It's actually got a topic reference to the Library of Congress. So a lot of publications are both can be topics as well as publication types. So we need to kind of make sure we're not getting those things confused. Literature reviews, review articles, there's lots of those.

It's not quite clear exactly what that's referring to. Case studies, but the type there is actually referring to a research method rather than a case study as a publication. Preprints, working papers, a case report,

that's a medical report, and scholarly articles. So it's just a sample of what we might be wanting to work on and create references for in Wikidata. So on the topic of climate change,

there are actually thousands of publications. So this is using Scholier to see what was available on Wikidata in terms of diverse kinds of content. And when you have a look at that, you've got a lot of scholarly articles and then a lot of review articles.

If we get rid of the scholarly articles and the review articles, we've actually got, on the topic of climate change, one report a year for the last few years and maybe two. So there's an incredible lack of representation

of the kinds of material that we would want to see on Wikidata about climate change that would be very relevant. On the other hand, a source like APO has got 1400 reports on climate change.

And I really sort of show this mainly just to confirm to people that the reports are out there. It's not that the reports aren't there and it's not that they aren't important and substantial research reports and this is the example that's given here from a climate change round table,

development round table, the Ministry for Environment from New Zealand. So we're not necessarily dealing with insubstantial unverifiable research. It's a lot of really substantial material. And also something about APO is to have a look

at the diverse kinds of documents that are also there, conference papers, discussion papers, working papers, strategies, briefings, articles, guides, fact sheets, policy reports. So there's a lot of diversity within this area.

So this diversity is kind of pretty problematic because it's never really been worked out. So there are various attempts underway at the moment to develop ontologies for different sort of genres.

And even the word genre on Wikidata seems to be used for literary kinds of content rather than all sorts of publication types which is sort of how they're more usually used. So there's another number of projects looking at this.

It would be great to see a project looking specifically at kind of report literature or organisation publications. I actually prefer not to use grey literature but I use it because it's understandable for a certain segment of the community. But if we think of organisation publications

I think it's the easiest way of thinking about it. So the Confederation of Open Access Repositories has made an attempt to create a vocabulary and that's referenced on Wikidata. Some of those are really useful.

That actually is interesting. A lot of these ontologies sort of refer cross-reference to each other. But so the core vocabulary reference is the GISG one from 2008. The Library of Congress has also got an extensive genre form vocabulary.

The Library of Congress sort of has a little bit of a bias towards books. In fact most of these ontologies have a bit of a bias towards books. So they often miss, the policy area

has a lot of publication types of its own special nature. Schema.org is very random I think in what it chooses to create a genre for. So that can be used to a certain extent. But yes, something interesting.

You saw the working paper definition in the Confederation of Open Access Repositories actually describes it as unpublished documents. It references the GISG scheme which actually describes a working paper as a published document. So even if it's referencing the same thing

it can actually have the opposite definition. So there's really quite a bit more work that needs to be done here. But having kind of talked about all the complexity and differences, we can absolutely look to books

as a guide to a lot of what we would wanna be using for describing a lot of these materials. They are documents, so they have a title, they have an author, the author is often multiple authors, often institutional authors. You need to think about the instance.

So it's a publication, a report, a working paper, a technical report. It could even be multiples of those. The title, the subtitle, the description. The publisher is often an organisation, often multiple organisations. So the organisation itself may need to be added

in the country. The publication date, that can be confusing. It's interesting to look at the IPCC report has at least one I looked at had two dates. This can be the case with government documents where the report has got a date

of say November of one year but it actually doesn't get released until February of the following year. The topic, so there are vast numbers of topics. So the topic is going to be quite complex and I think that's a big issue

that Wikisight needs to address is how we look at getting some sort of topic, vocabulary that we can use. The URL to the full article. So this is why I asked the question,

where do we go to and what's going to happen? A lot of this material is published online by the organisation and it doesn't remain stable. So this is a big problem. The full work where that might be available, various identifiers. So a lot of reports do have identifiers,

a lot of reports don't. Some that might have multiple identifiers. I have tried using identifiers to put things in and sometimes it works and sometimes it doesn't. So even with an identifier, it might not pull any content. The language is a part of a series of collection, et cetera.

So there's a really strong basis to develop a data model for different kinds of documents based on books, but we also need to think about the differences. So this is the example where it's actually got two publication dates.

I'm not quite sure why, but it's interesting that that's possible. You can also sort of say that something follows or is followed by. So this is part of a series. So it's the case with the IPCC. It doesn't actually have any topics on it for this one. So that's a real shame.

Another example, the Finch report. So this is a good example where it's really important to put the alternative title. So although the name of this report is Accessibility, Sustainability Excellence, it's generally known as the Finch report,

which is the case with a lot of major government reports. And I, yeah, so this is a general guide. Just be, yeah, oh, I'm sorry. I was gonna, I always looked at the link for the Finch report, and it's not a dead link.

It goes to, it's a moved link. So the content for it has been moved. I'll just, if I can do this. Yeah, so we go to the live page, this, it's got a link here. This actually goes to this page.

So it's still sort of, it's still a page. It might not register as a dead link, but as you can see, there's no copy of the Finch report. And in fact, you know, you try and look it up and well, you think you've got it, but actually this is the government's response to the Finch report. And from what I can gather,

gov.uk doesn't have a copy of the Finch report. You can find it on this website. I'm not really quite sure what this website is about, but anyway, they do have a copy of the Finch report. So, whoa, fantastic. This, so SemanticScholar has also got a copy of the Finch report, which apparently has got a DOI.

Oh yay, fantastic. Yeah, no, DOI is not found. So this is a wild west of verifiability for something that is, you know, like an extremely important government report

in terms of open access policy. So I'll just finish up. And am I going for time? Probably going over now. So yeah, just, we can start to sort of think about what kind of standard guide we could put together

to help people catalogue these sorts of materials. I'd be really interested in working on that. There's a lot of challenges in this area. So multiple departments name changes. There are a huge problem with machinery of government changes in government.

There's often multiple organisations, often, you know, whole groups are mentioned. It's really hard to work out who actually are any of them. Authors, there's often no identifiers. There may be multiple identifiers. There may be identifiers that actually don't even work anymore.

Copyright is often missing. So it would depend on the jurisdiction you're in as to what that means. In Australia, that means automatically all rights reserved. You know, you can have different copyright jurisdictions depending on something that's published or unpublished. So we've only just in the last couple of years

got a duration for unpublished works. And there's often missing information. You know, no date, no author, no publisher, no location, no identifier, sometimes not even a title, it's quite incredible. So yeah, just have a think

if you're ever publishing online and to make sure that, you know, it will float around by itself. So, you know, there's a lot of potential here. There's actually such an enormous amount of material that we really need to be looking at bulk import

and how to be doing that. A lot of organisations like the UN, the WHO, World Bank have very well organised collections. They're not necessarily all that great for doing systematic review searches on. And so actually there's an enormous need

for a large scale database where you can systematically review the literature on various, on any topic. And being able to do that and include the major research reports is a serious issue. There is no large scale collection

doing that at the moment. There's lots of small ones. There is a new, I should have put it on, there's a new project called Policy Commons, which I'm sort of vaguely involved with. And so that's one option coming along, but it's a real opportunity for Wikidata and Wikisite if we can work out how to do it at scale.

Then the, you know, institutional repositories have actually often got some level of content that might be able to be mined. The Bridge Library has a really great collection of reports. So there are various databases,

but they tend to be small to medium sized and having sort of, you know, a certain segment. So there is a real need for an overarching coordinating collection in this space. Okay, sorry, I hope I didn't go over time. I think I might've gone over time.

Don't worry at all, that's totally fine. As mentioned earlier in the session, we've unfortunately had a no-show from one of our presenters. So we are reshuffling the presenter order currently. And we've actually got a couple of questions for you, Amanda, that have been added into the etherpad.

So the first of those was a definitional question, actually. What do you reckon is wrong with the Wikidata definition of discussion paper? Oh, so I have to look it up again. I think it was that it, yeah, sorry.

No, my finger's not going to do anything. Yes, sorry. I think it was that it presumed to be coming

from a very much a research angle and wasn't, sorry, my computer's not finding it for me. No, sorry, I'm not getting it up.

Yes, sorry, that it was coming from the perspective of a research discussion paper, like preliminary findings, that sort of thing, whereas actually a discussion paper in a political context is often,

or in a research sort of public discussion context, is a lot more about kind of presenting ideas and policies for feedback. So just the nuancing around a lot of the definitions come from an academic perspective

and don't really take account of a policy or an organisation perspective. And there was actually an additional question about the APO taxonomy. So it asks, has the APO term taxonomy changed?

The link that the questioner had to climate change now returns a 404 error. So they add the link in the etherpad. It may have, Dave, actually.

So I haven't worked there for nearly two years, so I'm not responsible for that anymore. And they have actually kind of created a new taxonomy a public policy sort of subset taxonomy, which has been added to Research Data Australia,

their taxonomy list. And so it may have been that they've been changed. Yeah, so that's all I can say. I also wanted to second the comment that you made about the difficulties in this wild west of verifiability.

I was working on a project that required me to look at a particular report a while back. And not only did that report, of course, not have a DOI or any stable identifier like that, it didn't have a stable URL that pointed to it.

And it didn't even have a stable title. So it was published, it appeared in multiple different places under different variants spellings of its title. And so in many ways, Wikidata is actually more vital for these sorts of items than it even is for academic journal publications and books, which already at least have some stability

in their site ability. And actually for a lot of the organisation published literature and reports and grey literature, actually Wikidata might end up being the primary, the primary location for collating that sort of information.

I think so. Well, I think, so my feeling is that there will be various small scale collections because of the niche nature of it. But it's really difficult to do any search at scale,

particularly like if you're doing a systematic review, you actually want to be able to, because you can do large scale searches of databases, what you really want to be able to do is separate out,

okay, we've got all the journal articles covered. What do we need to find out that's not a journal article? How do we actually search everything that's not a journal article? That is, you cannot do that on Google, you can't do that on, or Scholar, you can't do that on Microsoft Academic. I think a lot of the databases

you can't really sort of differentiate from, I don't want to see journal articles. And then when you do get good big collections like the WHO, you can't run specialist queries on them. So I had a project last year looking at digital health,

and I was given this sort of, you know, string of 10 different terms that I was meant to sort of run on various aspects of digital health that you can put into like a commercial academic database or some of the sort of systematic review software systems.

Every time I would run this string on another database like WHO, I'd get nothing, absolutely nothing, because it just couldn't, you know, you had to actually go back to just find digital health, no any other extra qualifiers. So then there's really a desperate need

for an overarching aggregator, but also one that's able to be able to have a lot more complex metadata, you know, around research methods and all that sort of thing that could sort of start to be added into records for different content so that that can be reused.

Fantastic. Thanks, Amanda, very much for that.