Hi, I'm Brian Bales, Coordinator or Unit Head of the International Nuclear Information System at the IAEA, and I'll talk about charcoal in Zambia and how we can use user narratives to create successful and equitable information services. So first a bit of background about INUS.

It was founded in 1970 as a great example of Cold War cooperation between the US and the Soviet Union, and it quickly grew and now has 132 member states which contribute literature in all areas of nuclear science and technology. It has information just not just on nuclear energy,

but also nuclear medicine like imaging and cancer treatment on preservation of cultural heritage objects using nuclear techniques, soil remediation, and so on. And whenever I see this video and looking at all the equipment that it used to require,

I'm reminded of the great investment that the IAEA and the taxpayers of the world made to create this information service. Of course, we no longer use mainframes or printed books in the production of INUS, and it was really designed for research by nuclear scientists of various kinds.

But in the last 10 years or so, we've opened it for the general public through INUS repository search, and here you can see INUS repository search, and I've done a query on the effects of charcoal burning on the environment in Zambia, and I've gotten some results. And I'll tell you why I've done that particular query in a moment.

So just some numbers so you can understand where we stand. By the end of 2022, INUS had 4.6 million bibliographic records. 2.17 million of those led to full text.

Over 600,000 of these full text PDFs were hosted within the INUS repository. Now also in 2022, we had 4.6 million repository page views. We had 2.9 million unique searches, 2.1 million unique visitors, and

4.4 million PDF downloads. Now I want to go further to explain that we're highly dependent upon Google. Google provides about 80% of those 2.1 million unique visitors. So people are searching Google for a particular keyword. They get a

link to INUS as a result, and then they come to INUS in the typical case. So where this presentation comes from, that I was looking at Google Search Console, which can provide you numbers and the search queries that lead to your site. And I started to notice that we would suddenly get all at once a relatively large number of

long and very similar queries, all from the same country, all in the same day, or a couple of days. For example, we would get, how is carbon-13 used in medicine? Or what are the methods for detecting heavy metals in water? Or in this case,

what is the environmental effect of charcoal burning in Zambia? So we can see here that query and several related queries that were received all at once. Now looking at this particular query, you can see that there were over 250 searches like that, where we were presented to the user in Google, and

over 50 people then clicked on the link to our repository. All of the queries were from a single country, that of Zambia, and you can probably suppose that this was from a school assignment. Now having lived a few years in the neighboring country of Tanzania, and having seen both the charcoal vendors and

some of the schools, I started to wonder, how is it that we are serving this particular community of users? I mean, how effectively are we serving that community of users? Now we can also tell from Google Search Console that the users were led to this paper,

which is a publication of the Stockholm Environmental Institute given to us by Sweden back in 1994, and this goes into great depth on the environmental impact of charcoal production and utilization in Zambia.

Given all of this information, we can then begin to construct some ideas about the users and their experience. First, I want to make clear that this isn't a user story, and maybe you've heard of that. That's a more formal thing that's done in Agile and Scrum software development, where you say, for example,

as a student, I want to retrieve information about a subject so that I can complete an essay. So all of these would be collected to develop the requirements. Instead, we're talking about a user narrative. A user narrative describes a particular person's fictionalized journey through the use of your intended product.

Aptly named, it reads more like a short story. Most user narratives are between 500 and 1,500 words, and Jack Dorsey, the co-founder of Twitter, before it was bought, of course, says, if you do that story well,

then all of the prioritization, all of the product, all of the design, and all of the coordination that you need to do with these products just falls out naturally. So we can begin to construct the user narrative. Let's say I am a 14 year old student in Zambia.

I was given an assignment at school to write a one-page paper on the effects of charcoal burning on the environment. of Zambia. Now a lot of this is supposition. We have to to fictionalize or suppose some of the facts in this case, but it will help us to realize the

the user community that we're trying to serve. Now when I came to this sentence, it said I used my, now I wanted to see, then with facts I took some research to know what kind of platform she used. And if we look at the International Telecommunications Union,

we can see that there are over 100 mobile cellular subscriptions per 100 people in Zambia. So cellular phone penetration is quite large in Zambia. The percentage of people using the internet is just under 20% or was just under 20% in 2021.

So we could suppose that she used her mobile telephone. Maybe she used a school computer that had access to the internet. Maybe she used one at home. It's possible. But the safest supposition is that she used a mobile

telephone to search for this information. So here's what we have now. I'm a 14 year old student in Zambia. I was given an assignment at school to write a one-page paper on the effect of charcoal burning on the environment in Zambia. I used my mobile phone to search for that phrase in Google.

I clicked on one of the links and it took me to a PDF on the Innis repository. So I thought that I would then take this narrative, you know, put myself into her place and try it on my own phone and see what the experience was like.

Now I did a search for that phrase, what are the environmental effects of charcoal burning in Zambia in Google. And I wasn't that happy with the result. Because we see here a link on the top position from MDPI, the publisher,

Influence of Charcoal Production on Forest Degregation in Zambia from 2022. This leads you to a PDF and that's that's fine. Our link, which is the second one, goes to Innis, but the title says SEI EED dash dash 33.

So it's not very user friendly. It says cited by 12, which is interesting, and that it has 95 pages, but the the text really doesn't tell you very much. If you click on that link, which is the second picture, you're presented with a PDF, which is very difficult to read on mobile.

Furthermore, you have to sift through the 95 pages of the PDF to get your answer. All you wanted was what are the environmental effects of charcoal burning in Zambia, and you're presented with this PDF that goes into great depth on the subject that you then have to sift through and derive an answer from. So it's not very user friendly.

So some conclusions that we can draw are that this audience of users, they really want an authoritative and citable answer for use in a school report, but that's delivered in a user friendly way. This audience, as well as those around the world, is highly dependent and more and more so on mobile platforms.

And we can conclude that the current configuration of Innis is difficult for such users, and that we in Innis are not really meeting their needs. So what are some solutions that we could do? Well, an easy fix for the title problem is to fix the PDF metadata,

because it appears that that's where Google was getting the report number, that user unfriendly title from. Now we may consider large language models. They may be a solution eventually,

though they of course have a famous problem with hallucination, with making up answers, with basically telling you what they believe you want to hear. We could convert legacy PDFs to mobile friendly pages. Maybe that's a possibility. I think that would be

rather difficult, because we could do it initially in an automated way, but these would require a lot of fixing, I would believe. And I wanted to say that this technique can be used for other underrepresented audiences, such as those with vision impairment. You could load a screen reader, for example, and see how your site

appears to a screen reader and see what the experience is like. Is it easy? Is it difficult? Is there a lot of nonsense that's being given off from your page? And sometimes that does happen with screen readers.

So let's look now at large language models and what they could do. As I said, that there was a large problem with hallucination, with simply making up facts. And one of the first, or perhaps the first, large language model to address this was perplexity.

I did the query, as you can see. What are the environmental effects of charcoal burning in Zambia? And you can see that it cited six sources, including Ennis, UNEP, and others. And we can see that it's given a pretty good answer, citing each of the

sentences with the source that it supposedly came from. So it says charcoal production is a major driver of deforestation and forest degradation in Zambia, etc, etc. This is a pretty good answer. One that you could use, or one that you could use

after verifying all the facts, after looking at the primary sources, that you could use in your school report. If we look at another one, the famous ChatGPT. ChatGPT, at least in version 3.5, does not cite its sources. And I asked it, what are the environmental effects of charcoal burning in Zambia? It did give a pretty good answer.

But then I asked, what are your sources for this? It said that it couldn't give me an answer. So, from my experience, you really cannot trust ChatGPT to not make up answers.

It still does that. You know, perhaps a sentence or two will be simply made up and have no basis to it. If we look then at Bing Chat, Bing Chat is based on ChatGPT, but they've enriched it by also giving sources.

And we can see those sources below. If you click on the plus four more, you can see though that at least one of the sources is Wikipedia. So, it's given a pretty good answer, but maybe not one that you could really cite in a higher level high school or university paper,

because of the danger of made-up facts. So, we can conclude that large language models may be a solution, but they really need to improve still the hallucination problem. Now, I did some some checking. I did some queries earlier on

with large language models to test how they do with more obscure facts, and they've really improved quite a lot in the last year and are less likely to do that. So, perhaps soon they will be a good solution for authoritative answers. So, another thing that we could do is to create a wiki. A wiki that's written by somebody knowledgeable in the area,

and that's vetted, and verified by other people knowledgeable in that area. In this case, we've created a wiki article on knowledge management, and knowledge management is a big topic in the operation of nuclear power plants, because you can imagine

these power plants were built in the same era, the technicians were all trained and hired in the same era, and they're all retiring at around the same time. So, their knowledge is retiring with them, and so we need to preserve the knowledge and pass it on to the next generation.

But this solution would be very difficult for something as specific as charcoal burning and Zambia. So, perhaps this isn't a good solution for this community of users. So, whatever solution, whatever technology that we use to fix it in the future,

the important thing is that we know that there is a gap. That we know that this is something that needs to be fixed sometime in the future, and when there's a technology that's suitable for it, then we can seize on that and fix this problem. And so, as we design the future in a system,

user narratives will play an important part in making sure that we're actually meeting the needs of the users. Maybe some audiences we can't fully satisfy at the moment, but our intent should be to meet the needs of as many as possible. So, I thank you for your attention.