of the project. Then, I'm going to give an overview of our data collection methods before turning it over to Elizabeth, who's going to discuss the data collection methods in more depth, especially the focus group sessions and the use cases we derive from those sessions, and the follow-up survey based off of those use cases.

And then we're going to wrap up with our findings from the survey and our next steps. So back in 2014, the Digital Library Federation's Assessment Interest Group, or the DLF AIG, began studying current literature on the use and usability of digital libraries, and out of that, we authored a white paper that was released in 2015 called,

Surveying the Landscape Use and Usability, oh, shoot, sorry. Use and Usability Assessment of Digital Libraries, so where we summarized our findings using three umbrella terms, user and usability studies, return on investment, and content

reuse. And we found a number of articles written about users and usability of digital library systems, but very little on how material is actually being used and how to evaluate the impact of these digital collections. So each one of these little umbrella groups split off into different smaller working groups,

and we called our project, Developing a Framework for Measuring the Reuse of Digital Objects. Last year, we were the recipients of an IMLS National Leadership Grant to study current assessment practices and generating use cases for developing a reuse assessment toolkit

in the digital repositories community. And I also want to take this time to thank the Digital Library Federation, who handled the administration of our grant, funding, and provided us digital, physical, organizational

space to work on this project, as well as emotional support. Additionally, as part of this grant, we assembled an advisory board, and we were quite fortunate to have the very knowledgeable guidance and expertise of Hannah Frost, Emily

Gore, Kristina Matusiak, Sandra Phoenix, Dorothea Salo, and Ali Shekari. So this is everyone in our project team, who includes Elizabeth, myself, Caroline Muglia from the University of Southern California, Genia Ogara from the Virtual Library of

Virginia at George Mason, and Liz Wolcott from Utah State University, and Santee Thompson from the University of Houston is our fearless leader. We are talking about this project on Twitter using the hashtag digreuse, D-I-G-R-E-U-S-E.

So please tag any tweets that you write about this presentation, or if you have any questions, please use that hashtag and then we can get back to you. And then also you can learn more about the project overall on reuse.diglib.org.

So before we can kind of understand how to assess reuse, we have to understand what we mean by reuse. And we've tried, for the purposes of this project, we've tried to articulate reuse by comparing it and contracting it with the idea of use. And I'd like to note that the definitions provided here are not final.

They're very much in flux, and it's part of what we're trying to do in this project is to come up with clear definitions of what reuse is and how it differs from just use of digital repository materials. And we welcome your constructive feedback or thoughts on these definitions.

We define use as discovering and browsing objects in a digital repository often assessed as clicks or downloads without knowing the specific context for the use. Some examples are web analytics, Google Analytics that capture metrics on initial access

like, oh, I just said Google Analytics, or the metrics that are supplied by your digital library software. So in contrast, our definition of reuse is how often and in what ways digital repository materials are utilized and repurposed and knowing the context of that use.

So metrics of what and how often your digital content is shared on social media, like are means made out of your materials from your digital collections, or analysis of imagery use using reverse image lookup technologies.

So again, these are very much working definitions to get the conversation started. So as I said earlier, we're going to concentrate on our focus groups and the follow-up survey. We held three rounds of focus groups, and each round of focus group,

although originally we only budgeted for two rounds of focus groups with two in-person focus groups at major conferences, Code4Lib and DLF Forum 2017. And two virtual, two sessions of virtual focus groups each round.

Luckily, we had extra money in the budget, so we were able to do an additional round of focus groups, one virtual and one in-person. And so focus group participation was largely by invitation. We tried to invite professionals from different types of cultural heritage and knowledge

organizations, as well as diversity of people. Of course, everyone on our project team is an academic librarian, so representation from this type of institution was still dominant. So now I'm going to turn it over to Elizabeth. Take you through the rest.

All right, thank you, Ayla. Okay, so most of the data we gathered related to grey literature and reuse came from our focus groups, and I do want to note that we weren't specifically asking about reuse of grey literature. It just emerged as something that we thought was interesting.

So we conducted each focus group with two grant team members, a facilitator and a note taker. We audio recorded the sessions, and we said in our IRB, and I would definitely never do this again, that we would not create exact transcripts of the focus groups and that we would destroy the recordings in 48 hours.

So after we finished each focus group, we were all jumping on airplanes and listening to the audio recordings and taking the best notes we could, and then we had to delete the recordings and just work from the notes. We used a qualitative analysis product called DEDUCE to code the notes from each session, and we based our methodology on the open coding method. And then our final goal with the focus group data was to be able to use it

to develop digital library reuse use cases. So up here I've got some examples of reuse that emerged from these focus groups. We reanalyzed the data in preparation for this particular conference to pull

out specific examples and issues relating to content reuse of digital library hosted grey literature discussed by the participants. So what I have here on the screen are examples of some of the actual examples of reuse of grey literature materials that were provided to us, as well as some additional features and functions that might facilitate reuse for digital library materials in general.

We go into these and others in much more detail in the paper that we wrote for the conference proceedings, so I'm not going to read all of these out, but I do want to highlight a few that we thought were particularly interesting for Greylit. First, participants whose repositories include data sets have noticed that they wish they could assess what content in their repositories was not reusable

in order to develop deselection criteria. They also wanted to be able to see when institutional data management plans referred to their repositories so they could assess deposit and eventual reuse of these data sets. Some repositories housed analog data sets and considered the transformation of these data sets,

either by other units in their institution or by outside institutions of those data sets into digital form to be a type of reuse, and they weren't quite sure how to measure that. Measuring the overall impact of institutional repositories was noted as being a particular need for administrators. One participant noted that institutional repositories were increasingly being looked

to as official repositories for what they called data bundles, which included images, hyperlinks, PDF files, and databases. Grant agencies were then using the institutional repository to verify grant applicant claims, and the participants did not know how to measure the value of providing such a service.

Several participants also identified what they called the problem of electronic theses and dissertations. A lack of oversight or culture of attribution were mentioned as problems leading to institutional repository hosted theses and dissertations with either faulty citations or no citations at all. These pose a challenge to digital library staff hoping to find citations

to their hosted material and institutional repository hosted theses and dissertations. And finally, participants have several times differentiated between scholarly and non-scholarly reuse of their assets, noting that both should be measured but should be weighed differently depending on institutional priorities and mission.

Similarly, the impact measures of institutional repositories versus digitized special collections and archives versus data repositories or grey literature repositories might all be different. The focus groups also served to provide additional examples of what the participants considered reuse that we had not previously identified in our foundational survey.

These included the following, which we have then extended with some illustrations applicable to grey literature. The first was repurposing metadata in aggregated consortial or membership repositories. Inclusion of data in an aggregated data repository like the UK Data Service could serve as an example of reuse under that definition.

Indexing records with abstract and record location information and then selling it on portable storage media, we thought that health data could potentially be an example of this. And building a database from opened, digitized vital records and then connecting those to living people, which is the sort of thing we see frequently

with genealogical data, such as a database like Ancestry.com. So from the focus group data, we developed 18 generalized use cases, which were divided into three broad categories, data collection, analysis and reporting, collection development, and privacy rights management and ethics.

The overwhelming majority of participants indicated the need for best practices on all aspects of reuse assessment and also implied a desire to better enable reuse of their digital grey literature materials. We've heard about this a few times already today, but recommending or even requiring the use of digital object identifiers and other standard identifiers can facilitate assessment,

especially for reuse, but we found both through our literature review of grey literature reuse as well as our own focus groups that many institutions are struggling with this. However, some opportunities for digital libraries with grey literature collections may be found in the increasing pattern

of making grey literature available via institutional repositories, which not only aids in discovery of the materials, but can also facilitate reuse and citation, since institutional repository software may be set up to automatically generate handles for each deposited item. Depositing into institutional repositories also helps address another discovery challenge

for grey literature, which is the lack of a major aggregator, meaning a potential source of reuse data that is available to digital repository collections, aggregated via services kind of like Google Scholar or ShareWork. Even though in this project we did not focus specifically on grey literature reuse,

these findings are important because now our research team can further synthesize the results to develop use cases, features and functionalities for our future reuse toolkit that are designed and included expressly for the needs of the grey literature and institutional repository communities. So after we completed our focus groups, we issued a final survey

where we asked participants to prioritize the use cases that we had developed from the focus group data. What we found was that respondents were most interested in the following. Understanding how their digital library content is being reused in a variety of contexts by various audiences, social media, instructional uses, scholarly works, genealogy,

digital humanities projects, just to name a few. Telling stories of impact with the reuse data that has been captured and tailored for specific audiences and stakeholders. Assessing quality and quantity of items reused to inform future digitization projects and priorities.

Enabling and encouraging attribution of materials from collections in various reuse contexts, including through sharing and reposting on social media, integration into classroom instruction, citing in scholarly works or through non-academic venues. And knowing and understanding who their digital repository users are

without violating user privacy. We also asked surveys respondents to rank preferred features of a reuse toolkit. Participants suggested some system features that could accommodate reuse measurement such as including right statements with all digital materials and auto-generated citations for digital library objects.

Guidelines on reuse of specific content types with special properties like theses and dissertations is needed. Also, codes of practice for meeting the needs of indigenous and or marginalized groups that may have specific reuse requirements is essential. While there were not any specific examples of grey literature from indigenous communities discussed in our focus group, cultural appropriation

and the need for general codes of ethics for use and reuse of indigenous and marginalized cultural heritage materials emerged as an important theme that could be relevant to grey literature as well. So we completed the grant at the end of June and we submitted our final reports to IMLS in September.

All of our reports and publications and conference presentations, and we've done quite a few of them now, are available in our OSF Open Science Framework repository. Links to that and more are on our website which we listed earlier and I'll show it again in just a second. It's reuse.ditchlib.org.

We published an article on our preliminary findings in Performance Measurements and Metrics Journal that just came out about a month ago and that article is open access so you can all get to it. Some of our colleagues are presenting this week on some other aspects of this project, actually I think tomorrow, at the Association of Research Libraries Assessment Conference in Houston.

And finally, we have applied for a second IMLS grant in order to continue this work. Our next step is to take the information we've gathered through our research and use it to actually build a reuse assessment toolkit. So wish us luck on that grant and be on the lookout for more info. Thank you very much.