Very good. Yeah, thank you so much for inviting me. Very glad to be here and talk about citations and WikiCite. We've been doing some exciting stuff on the Wikipedia iOS app. Some of you may not know that we have an iOS app actually that's built by the foundation

where we get to do some experimental work that sometimes we can't bring to web. So in that vein, we've been working for the past year on three experiments. I'll talk about one of them today though. And these experiments are all around closing the

editing loop and breaking down the editing wall with the goal of helping readers to gain an understanding of how Wikipedia works without losing their trust in Wikipedia. The real intended goal here of this work is to empower readers to make corrections when they see mistakes and even potentially become really successful

editors and contributors in their own right as well. I've shared my slides. If you want to check out the additional experiments, feel free to tap the link in the etherpad. But the experiment that I'll focus on today is the featured references experiment. I'll go into some details about that as well. And if you want to follow along on our work, there's a fabricator ticket and two prototypes that you can play around

with on your own time as well if you'd like to. The user story here for this feature, which is our kind of guiding principle as we move forward in figuring out what we want to do, is as a reader of Wikipedia, I would like to know more about the citations used in the articles that I read so I can learn about how

Wikipedia is built using reliable sources and explore similar articles or sources. So, how do we do this? This is our first pass. There are two parts to this experiment.

One is an interstitial card that shows up in the article itself. This is our way of introducing readers to this experiment. We need something that sort of is visually significant so that it stands out when you're looking at a Wikipedia article. So, the way that we're focusing on doing that is by exposing sort of some of the

background chromes. So, it looks like, you know, the goal here is to make it very obvious that this was injected into the article and not necessarily something that was curated by editors. From there, there's a little bit of information. I'll go into more information about what's on this card. And should the reader decide to interact with it more, we will open a modal,

which will enable readers to learn even more about this featured reference. Before I go into things too deeply, I do want to talk a little bit about reference selection criteria. Again, we haven't actually built out any of this. We desperately need your help in order to have the data that we would need in order

to build this feature. But our proposal for the purposes of this experiment is to start with book references on English Wikipedia. The idea of utilizing book references is just because we would have a fair amount of structured information that we think we

could potentially use from some external sources like OCLC, but also just because we are simply running these experiments in English first. That being said, we are looking for feedback from folks around the globe who speak many different languages. So if you feel so inclined after watching this to send me some feedback,

I would love to talk with you. For the purposes of this experiment, there would be a maximum of one featured reference per article. And our selection criteria we're thinking would be the most cited book reference

in that article. And if no book references are cited multiple times, then we simply won't show a featured reference for this experiment. And the real goal is to create a template that can be used, modified, or removed by editors to insert a featured reference into the article. So we would be building out this experiment when we do as a template so that editors would have

control over it. But we would just put some in to begin with so that we can get some feedback from readers. This also brings me to some changes that we've made recently on the iOS app and the Android app as well with how we lay out citations at the bottom of the article

and references at the bottom of the article. And so there have been some typography changes, some updates to let you know when the link was last updated, and some new things there. So please do check it out and send us some feedback if you would like to. Moving on specifically to focusing on this featured reference experiment,

I'll go into a little bit of information about the interstitial article card. So this is sitting at the end of a section, and that's where we would most likely place this interstitial card. And it would show up wherever the reference was first cited in the article.

The interstitial card has some specific information. It has the title of the reference, the reference author or editor, the published date, a short description, which we believe we may be able to grab from OCLC. And then a wish would be that we could potentially link

out to places like HathiTrust or archive.org, other places where folks would be able to immediately read and reference the citation, this featured citation, or potentially even hook up to local public libraries to enable readers to get their hands on these resources.

Tapping on the card, though, however, would lead you to the detail page, which is the modal, which I'll show you now. So this is the detailed view, which contains more information about the

reference from both on Wiki and off Wiki sources. Again, if it's available, there'd be some borrow page, which is to be designed, which will list a bunch of places where the resource can be accessed online or at a local library. At the top, there are also stats about this

reference. So you can see how many editors have cited this work, how many times the work was cited in this article, the number of languages that are citing this work. That's something that we found a lot of readers don't know is that each language Wiki is different. And so we're trying to bring more awareness and reader education there as well. The about

this reference section also showcases the references impact within the article and across Wikipedia. And then at the bottom, users can also explore other articles that utilize this reference or see more in-depth information about this reference as well. So there's a link

out to other articles that are utilizing this reference. And then even further down, we have... I'll find one where it's not moving in a minute to show you. Actually, I can go back here where the subjects are turned into... The subjects from OCLC are in fact turned into links

out to those Wikipedia articles. And there's some other information as well, structured information. Great. Moving on, I can just show you what interacting with this experiment would feel

like. So the reader is reading the article, they find the end of a section, they then are able to tap in and see this model where there's more information for them to explore as well as other articles and other subject matter. And they can link out to lots of

other stuff too. So that's kind of what the look and feel of this experiment would be like. We have done some exploratory surveying to see how folks are reacting to this idea. The first survey was sent out internally to WMF staff via the foundation optional mailing list,

and that's the information that I'll be sharing for this talk. But essentially, the 15 WMF employees completed the survey and these are their findings. Additionally, though, we did have 12 participants who filled out the survey from Tech News, as well as 26 iOS mobile testers participated in the same survey. I still need to go through

those results so I don't have them for you quite yet, but they are very similar to the internal findings. So some of the high-level findings as far as the featured references experiment went was that most participants felt the experiment was something that they could see themselves using, and they believed that it would improve trust and reader understanding

in how Wikipedia works. What was at top of mind for many of the respondents was about the placement and the selection criteria. People had a lot of questions about how we would do that. And everyone wanted to know where the information was going to come from around the featured reference selection. So this is where we would love help from WikiCite. We really need

a way to get this information and a lot of people are really thirsty to use this information in their products and features as well. The one piece of negative feedback we had was around utilizing editor count as an indicator of letting a reader know how much time an editor

has spent in the projects or sort of their commitment to editing. So it could potentially be a little out of place or a negative indicator. So we'll have to work on other ways to sort of show trust for editors to readers who are not really in the fold quite yet of

understanding how editors work and why people have interesting usernames and things like that. There was some good feedback as well. Most participants felt that, yeah, they could see themselves using it. A bunch of participants had this idea that this could be sort of a rabbit hole

for exploring other articles and resources. So it'd be another way to sort of dive deeper and focus in on one's own explorations and questions and things like that. And generally though just having a greater focus on references was seen as really positive. There were some questions

and concerns. People were excited about the prospect of borrowing the book but they had no idea how this would work. We are also trying to figure this out. One participant felt unsure of how to navigate back to the articles. We're working on how to make that modal transition

more clear. Six participants wanted to know more about the selection criteria or how editors could change or remove the featured reference and this is something that we'll definitely build with in mind, making sure that editors have the real say in what is shown on the article. A few participants had concerns about placement, this feeling that the interstitial is too large or

prominent or high up on the page. That being said, for the sake of an experiment we really need eyeballs on here but we can become more and more subtle as we refine the future. And then, yeah, again just this thought that maybe editor count could potentially be a negative indicator. There were some requests as well, related Wikidata items for any book or journal

being cited would be very cool. Some transparent information around selection criteria, so having some sort of information page, information about why this reference is qualified to be on Wikipedia which I think is really interesting so we could have some other ways of educating

readers on what we use as qualifications for references or what make a reference really powerful on Wikipedia or respected. And then just a way to browse similar resources. So yeah, summing up, most people felt like they could see themselves using this feature if it was available in the app. There were a couple of maybes which seemed more like yeses and then

one no. Most participants again felt that this feature would help improve their trust in Wikipedia content. Interestingly, when we tested with external participants, people felt like they already had such a high level of trust that they couldn't quite imagine

having even more which I thought was kind of a nice moment to reflect on but of course we can always do better so I think it's good to strive for that. And then improving an understanding of how Wikipedia works. Yeah, so again these are internal participants so they're sort of guessing around how a reader may feel but yes, most people felt like this is

something that would potentially improve understanding of how Wikipedia works for readers. And that's what I've got. So thank you so much for your time. I would love any feedback or thoughts that you may have. Feel free to email me or come follow the iOS project

on Fabricator. Thanks so much. Oh Amir, I can't hear you. Sorry. Yes, of course this happens

all the time. Thanks a lot for coming. Thanks a lot for presenting. There are positive comments on YouTube and relevant questions so people want to know whether it's possible to find information. So you mentioned this and I see that Liam already liked this and people are

saying that the detailed view is beautiful. So yeah, great. I agree. It isn't too beautiful. Not just because we're colleagues but because as an editor, I've been with Wikipedia for many years and I do think that as an editor I'd be happy if readers saw this this way,

emphasized nicely. There are some interesting questions in the chat. So this one is by Shivan Leachman. I hope I pronounced the name correctly. If no book references I cited multiple times then no featured references will be shown. Is a book always the best reference? In the

biodiversity field it is often an article. So I think I know the reason why you chose books but I'll let you reply. Yeah, so this is just for the experiment and thank you for the question. For the purposes of the experiment we're trying to stay a little scrappy and we were

hoping that we would actually have enough data at hand to build this experiment. We're not so sure about that in this exact moment. So the goal was to utilize since with books we would potentially have access to OCLC's API we could pull out more information that way. I think it would be a real shame if the only featured references were books. That would really be a

huge disservice to many topics that are popular on Wikipedia and that our readers want to know more about. So this is just for an experiment but we would definitely love to have articles or websites or all of the different citation types be possible featured references as

well. Yeah. Okay, so one more question from YouTube and then I think we'll go to the next presentation. This one is from Matthew Slager. Any thoughts of using Wikidata for quality rank sources so I as a reader can select what sources I would like to see displayed or get

an indication of how good the source is? So Wikidata has a qualifier at least for some things like the verification level qualifier. I don't remember the names but yeah like I think it's choice best or something. I don't remember but they have quality. They have

preferred normal and deprecated. I would suggest that is not the intended purpose of that feature in Wikidata. That is a way to indicate an individual statement a particular fact is the most

current fact or is an out-of-date fact or is actually a wrong fact that has been nonetheless used in a verifiable location in the past. It doesn't register the value of the source as a whole and it certainly doesn't really measure the value of that source

compared to any other source which may or may not be used in other theoretical. The reference itself can be deprecated if their reference is not good. Not the fact but

only the reference but I understand that mechanism is not for that. Yes, thank you. Admittedly this is not something I know very much about so if you have ways that you believe that Wikidata could become a part of this feature please do reach out. I'm always trying

to learn more. Thank you. For that particular idea, the editorial layer I would suggest is required because it's not in the structured data. The idea of a featured reference is not

necessarily a structured data fact. It's an editorial decision. This is the footnote I consider to be important in this article for whatever choice as an author of prose. So if it were to be my opinion, my first guess is if that were to become a feature, an accepted feature, a useful feature used across a lot of Wikipedia that might require a new metadata field

in the article to say footnote number five, that's the one I want to highlight and that would be something that the community would discuss and debate on the talk page or whatever, not as something that would be automatically generated by a heuristic of the most repeated

footnote or something like that. But that's a manual process that would need to be introduced as a community editorial decision, I guess. Or maybe in Wikidata we can have a number of situations for specific sources to have some hints for the quality of the source.

Just one really tiny question for me before we go to the next presentation. So all of these

things, does your team, the engineering part of your team, do they know that they have all the technology in the MediaWiki platform, in the articles, in the template

player, whatever, to actually implement this? Or will there be needed any big platform changes to actually get this done? And if you already have this, then do you actually have a plan to implement this and deploy this in the foreseeable future and do you have any guess of when will this happen? So no, we don't have all the information that we need to build this. We've

been talking to some of the infrastructure teams in order to see, and other partners spoke to Liam and we're very excited about WikiSight. But essentially, no, we don't have a way to know which references are linked to which articles in a structured manner that we would

want in order to do this. So some things still need to be built and having structured references would be really huge and would be the real push that would make this able to be real. That being said, because it's an experiment, we can be a little bit scrappier and we may end up

doing a quick A-B test at some point around this where we essentially just use tables that are better manually curated in order to get this to work. So that way we could hook in an API or something else later, but this way we would be able to get feedback from readers and from

the community with sort of this, not dummy data, real data, but just hand curated to make it work for us. But yeah, we need help. So if you have an idea of how we can access structured

citations and get more information about references in a structured manner, we are all ears. But please do keep your eye on iOS for around the next year. Probably we'll still be rolling out experiments. We have three. The first one is around, it's called articles living document, and it essentially brings aspects of the history page onto the article itself and

creates a more reader friendly view of history that's going out very soon. It's already in the beta version of the app as well. And then we have one more, which is an inspection mode, which is based off of the work that community detected with the, who wrote that tool, which

we'll be working on next. And then we'll work on this featured references after that, once we hopefully have more information. Thank you. Carolyn, could I ask you to put the links to those programs and to the fabricator board or the relevant place to follow up on these things

into the etherpad? Yeah, definitely. So people can follow up. Sounds great. I'll do that right now. Thanks so much. Thank you.