Hello, my name is Oller Tullen and I'm a PhD student from Hanover, Germany. Today I will present crowdsourcing scholarly discourse annotations. Let's get started. Scholarly communication is largely document-based. For example, when reading articles we are often faced with PDF files.

While document-based communication generally works fine for humans since it's created for human consumption after all, machines cannot easily parse the contents. Or in other words, the articles are not machine actionable. And one of the consequences is that finding relevant articles becomes more cumbersome

as more articles are getting published. In this example, more than a million results are displayed in Google Scholar when looking for user interfaces based on machine learning. When knowledge graphs are used, it is possible to create paper descriptions that can be understood by machines. This makes it possible for machines to understand the semantics or the meanings of the article

and for example provide the ability to have better support for finding articles. In this example, a paper is displayed. In grey, you see the metadata of the paper and in red some of the contents in this case an annotated sentence with a respective discourse class.

Having structured paper data has many benefits. However, creating this data is not that straightforward. Broadly speaking, there are two approaches. Firstly is to automatically generate these descriptions. For example, using natural language processing techniques. But this is not very accurate currently and often requires domain-specific models

and they are not always available. Secondly, there is the manual creation of structured data. For example, using crowdsourcing. Compared to automated techniques, the accuracy will be higher but since human labor is involved, scalability is an issue.

Also, it requires skilled users to make decisions regarding the modeling of the data. In this research, we suggest a hybrid approach to generate these structured paper descriptions using a paper annotation system. This uses both machine intelligence and human intelligence and it tackles the weaknesses of the techniques when they are employed in isolation.

It even provides a synergy in the form of an intelligent user interface. To create such a hybrid system, we will answer the following research questions. How to design an intelligent user interface to populate a scholarly knowledge graph using crowdsourcing? And a second question, how to employ a machine-in-the-loop approach to assist users in this process?

Let's now discuss the system design. This is an overview of the system design. The interface is displayed in the middle and the use cases for generating structured data on the left and for using the structured data on the right. The workflow is as follows. A user or a researcher uploads a paper in PDF format to the system.

Afterwards, the user annotates key sentences within the paper. Each sentence is annotated with a discourse class describing what the sentence is about. Based on the annotations, a knowledge graph is created. Let's now have a look at the individual components.

First, let's look at the data entry use cases and specifically the paper submission use case. This describes a scenario where researchers can annotate their papers during the paper submission process. This could, for example, be during the camera-ready submission. In this case, the author is of course familiar with the paper's content

and can therefore make good decisions on which sentences are most important and therefore which sentences should be annotated. Now let's have a look at the user interface. In the interface, you'll see the article is displayed on the right side and the annotations with their discourse classes on the left.

Now we'll discuss the three machine assistant components starting with the automatic sentence highlighting. The first smart component automatically highlights potentially important sentences in the paper. For that, the paper is first automatically summarized using an extractive summarization tool.

This created summary is split by sentences and each of those sentences are highlighted in the PDF file. Here you can see the highlighted sentences within the paper. And here you can see a button to disable the highlights in case they are not helpful. Highlights only serve as suggestions and it's therefore not mandatory to use them.

Next, the maximum sentences per annotation. This component counts the amount of selected sentences for a single annotation. A warning is displayed if more than two sentences are selected. This encourages users to only select key sentences and not to annotate whole paragraphs. Also, this warning is only a suggestion.

It's still possible to annotate more than two sentences. Next, the automatic class suggestions. Once a sentence is selected, a set of potentially relevant classes are displayed. These classes are determined using a zero-shot classifier and are based on the contents of the selected sentence.

Also here, in case the suggestions are not helpful according to the user, they can be ignored. Now let's have a look at the human role in this process. A user, of course, has to select the sentences within the article and then choose the relevant discourse class. They can choose from 25 classes which are coming from the Discourse Elements Ontology, or DEO.

Finally, there are multiple use cases related to how to use the generated knowledge. This includes an improved search system or automatic generation of paper abstracts. Now we will look at the demonstration of the interface. This is the annotation interface.

The first thing we do is to upload a paper in PDF format. Then we can start annotating a sentence by making a selection. We can either select a type from the list of all classes, or we can choose one of the suggested types.

In this case, the class problem statement seems to be right. After adding the annotation, it is displayed in the left sidebar. The completion bar indicates if there are at least two annotations for the five most important classes. We can enable the smart sentence detection, which will highlight potentially interesting sentences throughout the article.

A highlighted sentence can be annotated by clicking on it and then selecting an appropriate class. In case more than two sentences are selected for a single annotation, a warning will be displayed.

This warning informs the user that it is better to annotate a maximum of two sentences. Finally, the result can be saved. Now I will discuss the evaluation of this research.

The main goal of the evaluation was to determine the task feasibility and see whether researchers are willing to annotate their papers with this system, for example during paper submission. In total, 23 researchers were part of the evaluation. They were asked to annotate a paper they altered themselves, or to annotate a recently read article.

Afterwards, they were asked to fill out a questionnaire. The questionnaire consisted of three parts. First, to assess the attitudes towards the task itself and the smart system components. Secondly, a system usability evaluation was conducted to evaluate usability.

And finally, the task load was evaluated using the NASA task load index. Now the results. The system usability skill results show a score of 76, which is considered good. The NASA task load index was 35, which is low compared to a meta-analysis related to this index.

And in this case, the lower the better. Finally, here are some results of the participants' attitude towards the interface. For example, most participants are willing to annotate a paper during the camera-ready submission. Interestingly, they were mostly positive about the integration of smart technologies in the interface.

Even though they indicated that both the smart type suggestions and the sentence highlighting tools were not always providing helpful suggestions, they still appreciated the integration. Finally, the conclusion. We conclude that machine-assisted paper annotation is a feasible task.

Researchers are willing to annotate their papers. Also, they appreciate the smart elements. The other evaluation results show the usability of the system is good and the workload is acceptable. The limitations are mainly related to the evaluation. For this, more participants should be included from more diverse backgrounds.

This is also something we are addressing in future work. Additionally, we will use other NLP tools, possibly supported by humans, to further process the annotated sentences. This was the presentation. Thank you for your attention and please let me know if there are any questions.