Day 1: Practical Applications of Altmetrics and Novel and Experimental Uses of Altmetrics
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 8 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/15221 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
2
3
00:00
Normed vector spaceExecution unitBellman equationDean numberNP-hardTwin primeMenu (computing)MIDIChi-squared distributionPositional notationVulnerability (computing)Graphical user interfaceLibrary (computing)Convex hullProcess (computing)Smith chartInclusion mapFrame problemDisk read-and-write headEmailRule of inferenceUniform resource nameHill differential equationLimit (category theory)Electronic data interchangeData miningPrinciple of maximum entropyOvalMaxima and minimaDrum memoryTunisAlgorithmMathematical analysisAttribute grammarCodeVideo gameDatabaseData managementDiagramFormal languageGraph (mathematics)InformationDrill commandsLie groupTouch typingMathematicsComputer clusterNatural numberOrder (biology)Perspective (visual)RobotFeedbackCombinational logicSelf-organizationSoftwareGame theorySpeech synthesisStatisticsTelecommunicationNetwork topologyValidity (statistics)Dressing (medical)Time seriesComputer programmingExpert systemFrequencyLibrary (computing)Type theoryDigitizingBuildingProduct (business)Profil (magazine)Wave packetLevel (video gaming)Perturbation theoryWindowElectric generatorCategory of beingAverageDrop (liquid)VideoconferencingResonatorTriangleSoftware testingIntegrated development environmentTask (computing)Subject indexingComputer configurationDecision theoryPrisoner's dilemmaTotal S.A.AnalogyConstraint (mathematics)Mathematical optimizationState of matterProcedural programmingAnalytic setArithmetic meanAxiom of choiceForm (programming)BitBit error rateComa BerenicesSeries (mathematics)Division (mathematics)Programmable read-only memoryUniqueness quantificationConnected spaceExpected valueFaculty (division)Forcing (mathematics)Formal grammarSheaf (mathematics)Line (geometry)Green's functionUniverse (mathematics)Group actionLetterpress printingIdeal (ethics)Price indexContent (media)IntelLimit (category theory)Knot theoryPower (physics)Extension (kinesiology)Materialization (paranormal)MereologyMetric systemMoment (mathematics)Pairwise comparisonMountain passPhysical systemPhysicalismUniform boundedness principleProjective planeSlide ruleExecution unitResultantSatelliteDot productCore dumpLocal ringMilitary baseTable (information)Taylor seriesTerm (mathematics)TunisVirtual machineVisualization (computer graphics)AngleCountingNumberTime zoneCellular automatonCentralizer and normalizerScaling (geometry)Density of statesElectronic visual displayEmailLink (knot theory)DataflowQuicksortAreaE-bookDigital mediaSystem callPublic domainSurgeryMachine visionGoodness of fitReal numberDivisorMeasurementServer (computing)Dependent and independent variablesPlanningCovering spaceText editorOperator (mathematics)Water vaporBasis <Mathematik>Parameter (computer programming)WeightHypermediaCASE <Informatik>Process (computing)CurvatureStrategy gameSystem administratorError messagePresentation of a groupMetropolitan area networkBoss CorporationInstance (computer science)Food energyField (computer science)Vector potentialCross-correlationPersonal digital assistantSquare numberExterior algebraShared memoryHand fanAdditionPoint (geometry)Vapor barrierRankingSocial classGenderCuboidMoving averageGradientSet (mathematics)Open setArithmetic progressionInterior (topology)DissipationWord19 (number)Reading (process)CircleWeb pageCartesian coordinate systemSpring (hydrology)Online helpData conversionObservational studyInclusion mapStress (mechanics)Direction (geometry)Latent heatAcoustic shadowInteractive kioskInternet service providerInsertion lossFocus (optics)Student's t-testSearch engine (computing)Lattice (order)Electronic mailing listData miningFile formatFrame problemReflection (mathematics)Software frameworkGreatest elementTraffic reportingGoogolGraph coloringAddress spaceSound effectSource codeEuler anglesPlastikkarteWorkstation <Musikinstrument>Sign (mathematics)OutlierMetreEvent horizonBit rateView (database)TrailWhiteboardWebsiteSelectivity (electronic)Endliche ModelltheorieCoefficient of determinationKey (cryptography)40 (number)Different (Kate Ryan album)AuthorizationOnlinecommunityMUDMedianNeuroinformatikBlock (periodic table)Computing platformTouchscreenBlogElectronic program guideInverter (logic gate)Archaeological field surveyPrincipal idealMachine learningElement (mathematics)Classical physicsCondensationCycle (graph theory)EvoluteElectronic GovernmentContext awarenessMultiplication signRule of inferenceFreewareDigital photographyStandard deviationOptical disc driveMessage passing2 (number)Program slicingMappingSpacetimeRight angleService (economics)Raw image formatOperating systemOffice suiteDemosceneFigurate numberPattern languageVolume (thermodynamics)Game controllerComputer-assisted translationMobile WebTwitterRepository (publishing)FacebookSoftware developer1 (number)RhombusBookmark (World Wide Web)Computer scienceFunction (mathematics)Phase transitionTemporal logicCausalityNumeral (linguistics)MultiplicationBroadcasting (networking)Similarity (geometry)Dean numberShape (magazine)CumulantMetadataComputer fileInternet forumNewsletterDisk read-and-write head10 (number)CurveData qualityWeb crawlerTesselationWeb 2.0Digital object identifierGoogle+Lecture/ConferenceMeeting/Interview
Transcript: English(auto-generated)
00:09
OK, there's some silence coming in. I think this is the time to start. So probably the group of social media lab people is coming in a minute. So welcome, everybody, to this next session,
00:21
Practical Applications of Altmetrics. My name is Hans Seelstra, project manager at Elsevier for metrics, journal and article metrics. And today, we're going to have some really great stories coming from people in the field. So really, how is altmetrics applied in day-to-day situations, and what can we learn from it?
00:43
And we have a great team of people lined up here. From kudos from the university library. And I think we're also looking at Sasha. Sasha is around here, too. Sasha Nukovic, he's the editor-in-chief
01:03
of STEM Fellowship Journal. I don't see him around right now. So probably he's also in this session with the social media lab people. So he'll hopefully be back in time. As we all saw, and that's always the trigger
01:25
after lunch, what we heard from Martin Kirk is how do we get from interesting to mission critical? Because that's, I think, really the essence, right? We've talked a lot, we've seen some frameworks coming along.
01:41
Yesterday, Mike Taylor was talking about how, in fact, so many articles do not get the attention they deserve. And there's a lot of articles going out, they are delivered after a lot of work, but that's it. They're not mentioned in social media, they're not even cited. They may be downloaded, but also there's
02:02
a lot of work to do for us. And I think in this session, we will see some tools that could be used and some real great case studies that should be applied in our daily work. Because so far, I think scholarly communication has failed to really engage scientists.
02:21
We have done not a great job so far. We can measure things, but there are still a lot of work to do in scholarly communications and science communications. And hopefully this will help in there, but we have to engage also with funders to make sure that things maybe really go more proactively
02:44
in funding requirements and stuff like that. So to start with this session, we have David Sommer. David Sommer is the founder of Kudos. There's the session coming back. Welcome, everybody.
03:01
Please join the crowd, including Sasha, one of our speakers. It's not your fault, Sasha, don't worry. It must have been very interesting, right? Tell all about it in a minute. We know the post-lunch dip is gonna hit us, right?
03:21
So everybody is probably willing to look at their screens and look at their mobiles and see what's going on. Please bear with it a minute. Please try to close down your laptops if possible. Please get away with your mobile. It's time enough to Twitter and blog about this.
03:40
That's no problem. Let's try to have some eye contact today. And the first person who's gonna try that out is David. He's not only gonna talk about how Kudos could help you in, let's say, spreading the word, but also he's gonna talk about fishing. So maybe that's a trigger for people who think, hey, I know Kudos already.
04:01
There's gonna be some new stuff in there. So please welcome David Sommer with a warm-up house. Thank you very much. Very good. Okay, so Hans has asked me to inject a bit of energy
04:20
into the proceedings, and the way the organizers are gonna achieve that is by having me stand this side of the podium rather than that side of the podium, so interactive. Thank you very much for inviting me to be here. 4 a.m. is a great name for this conference because with the jet lag from flying over from the UK, 4 a.m. this morning is the exact time I woke up ready and alert. So there we go.
04:41
So I'm gonna talk about opportunity indicators. I'm also gonna talk about fishing, and I'm gonna talk about pigs as well. So all of that will be covered in the next 10 minutes. So opportunity, what do we mean by opportunity? So I'm gonna pull together three themes here. By opportunity, I'm talking about the potential, potential for good things to happen. Whatever that might be.
05:03
Attention, we're all very familiar with attention. By attention, we typically mean is something resonating with an audience? Are they actually connecting with whatever the message is that you're putting out there? And then the final part is effectiveness. So are you actually achieving the end results that you want to achieve?
05:21
So if we pull together those three things, opportunity, attention, and effectiveness, let's have a little sort of review of how we're doing on these three areas. So opportunity I mentioned is potential. You can think of that as the audience. So let's make a really simple analogy here. Let's think about tweeting. So if you are someone with a high number of Twitter followers,
05:40
then you have high opportunity potential. You have a high reach within your networks. So it's very much about reach and audience. Attention is about awareness. It's about is something relevant to you? Do you take action? So back to the Twitter analogy, that might be somebody clicking on a link in a tweet. And finally, effectiveness. Are you getting the end results
06:01
that you want to actually achieve there? So typically that's measured nowadays by citations, downloads, views, reads, and so on. Be interesting to see where those measures move on to in the future. So how are we doing? Well, if we look at attention, so we've got great companies like Altmetric.com, Impact Story, and Plum, who are doing a great job at providing attention scores,
06:23
and Qdos currently works with Altmetric.com. That's great. But as we heard this morning, there are a number of places where attention is happening that's not getting captured. So we've got things like ResearchGate, we've got these silos, academia.edu, SciHub, SlideShare, LinkedIn,
06:40
who no longer have an API that is included within Altmetric scores. So there is a lot of attention happening online that's not being captured. There's also a lot that's happening offline. So speaking at a conference is a great way of capturing attention, delivering your message, but that's not gonna come up in any sort of score necessarily. So how do we actually complete the picture there?
07:00
In terms of effectiveness, well, citations, although we hate it, that is the currency of research for many. And Clarivate, Scopus, Google Scholar, doing a great job on providing citation counts. CrossRef also doing a very good job on citations. We've been running some experiments recently to compare CrossRef time cited data
07:21
with other providers. And actually CrossRef is doing a very, very good job on that. And Counter, of course, the standards body for usage, that is very much an established method of looking at effectiveness. But it's the opportunity area that is missing. There's not a lot of work going on to actually look at the potential
07:41
for communications to reach their goal. And that's very much where Kudos is focusing its time at the moment. There was a great saying from a bibliomatician from Loughborough University in the UK who spoke recently, and she said this fantastic phrase. She said, weighing a pig doesn't make it any fatter.
08:02
No matter how many times you weigh the pig, what scales you use, what metrics you use, weighing it does not make it any fatter. The way to fatten the pig is to understand what makes pigs fat and then feed the pig. So Kudos is very much about fattening the pig. Very important to use the metrics to understand the before and the after,
08:21
has it actually worked, but actually just measuring it by itself doesn't actually increase the weight of the pig. So as a researcher, what you generally want is focus. How do you focus on the actions that are gonna generate the results that you want to see? There's lots of things you could do, but which ones are actually gonna lead to the results that you want to achieve for yourself?
08:44
And if we look at the classic sort of triangle of time, resource, and results, so you could imagine, let's say that you are a UK researcher and you've got to achieve a certain level of research for the REF, the Research Excellence Framework on which your institution's funding depends. So perhaps you need to achieve a certain result.
09:03
You need to increase from a three-star average to a four-star average in your REF rating. And you've got limited time to do that. So time and results are fixed. How do you best use your resources to achieve that? And that's not just about where you should share and channels and so on. It's about being really creative.
09:20
Should you make a video about your work? Should you have a cartoon abstract? How should you get really creative about that? Or perhaps your results are fixed and your resources are fixed. You have a limited budget for outreach. You have limited availability there. How could you best use your time? Which actions are gonna make the best use of that time and which ones aren't? And I'll show you a mini case study
09:41
around that in a moment. And finally, if you've got fixed time and fixed resources, then how can you get the best possible results for your efforts? So it's all about guiding researchers. What is the best thing that they can do with their limitations in terms of time and resource to get the best results? So a little mini case study on this.
10:00
This is Toby Green. Many of you will know Toby Green. He's the chief operating officer at OECD. And Toby wrote an article that was published in Learned Publishing back in August all about Sci-Hub and about how we failed and black open accesses trumping green and gold and what we can do about it. So a really, really good article. I recommend you have a read of that
10:21
in Learned Publishing. Toby is a big fan of Qdos. So what he did was he used Qdos to explain his work in plain language. So this is our publication page on Qdos. So he wrote a short title. He wrote what's it about in plain language, non-technical language. He wrote why it's important and also gave his own personal perspective on that.
10:42
And he took about 10 minutes, something like that to actually write that plain language summary. What's interesting is what happened next. So he also shared this and Toby shared it quite a lot. He shared it 21 times in total and he shared it through Qdos. And what that lets us do is map the actions he took
11:02
with the results he generated. So kind of not causation but certainly correlation. So this is his results and he's given me full permission to share this with you by the way. So you can't really see on this small chart, I'll make the slides available, but the little A's that you see there are when the author took an action such as sharing or adding a plain language summary.
11:20
Every time he shared, you can see the numbers bumped up, both the full text views as well as the share referrals. The altmetric scores also followed the similar pattern there and it now has an altmetric score of 419. So he shared 21 times. That resulted in almost 2,000 clicks on those shares
11:40
and almost 3,000 views of that plain language summary on Qdos with about 500 click-throughs to the full text from 21 shares. So that's a fairly sort of short amount of effort in terms of time going into generating those results. These are the actual shares he did. You can see the different platforms there, LinkedIn, Twitter, trackable links and so on.
12:02
And the one I want you to focus on is that one which says 1,334. So one of Toby's shares generated 1,334 clicks. What's interesting is where that was actually shared to. So he created a trackable link and then posted that to liblicense
12:20
which is the listserv discussing library licensing issues and so on. Many of you will be familiar with that. And Toby couldn't quite believe the numbers so he talked to Anne Oakerson who runs liblicense and Anne said there are about 4,000 active subscribers who received the liblicense email. So 1,300 out of 4,000. So that one post to liblicense generated a 33% response rate
12:45
of its potential audience. That's quite a good use of his time. But if you look at some of the other channels there, so LinkedIn for example, he shared three times to LinkedIn and he got three clicks, three clicks and zero clicks. So six clicks in total from LinkedIn.
13:00
So for Toby, knowing where to post is really, really useful information for him. Liblicense, really good source for him. LinkedIn, not so much. Twitter, better than LinkedIn and so on. We all like a framework. So here's a framework that we work with which is Dr. Pat Bazely's framework on research performance. And if you think about it, the top half of that
13:21
is very much about doing the research. So it's the methodology, it's about how you practice your research, how you engage with your colleagues and so on. The bottom half of that is all about communicating the research. So this is about publishing, this is about sharing, social media and so on. And it's that bottom half where we're very focused at Kudos.
13:40
And I want to define a couple of measures that we can start to sort of play around with here. So if we start thinking about what we call reach, resonance and result. So reach is similar to opportunity. Reach is very much the measure of an audience size. It's the potential for something to generate impact. Resonance is whether those shares,
14:00
those things that you're doing are actually having the effect you want to have. And then results are the end results of that, the overall sort of end goal of all of this. And there's also a thing called attributes, which is properties of an organization or a network. So for example, if you're at Stanford University, you might have a higher attribute score than somebody at a different university,
14:20
and also the quality of your network. So we can take those four things, reach, resonance, results and attributes with this very big data set that we're gathering, and then we can provide custom recommendations to people like Toby. So we can say, this is what you should do next. If you've got limited time, this is the one action based on the evidence that's gonna have the best result for you. Whether we serve that up through action cards
14:42
where you get your three cards a day, or recommended actions, percentage completes and so on. So this is a very kind of simple case of this. We're expanding this using machine learning and AI to really improve the quality of those recommendations. But it's evidence-based, that's the thing. So does attention or can attention drive action?
15:01
Can it cause results? We did an experiment with about 10,000 articles. We divided them into two sets, normalized the journal, age, and so on. So they're very sort of similar sets. For about 5,000 articles, the authors had used Qdos to explain and share their work. And for 5,000 articles, the authors had not used Qdos.
15:20
And what we were interested in is whether the full text downloads for those articles were noticeably higher if people had used Qdos. And we were really pleased to say, yes they are. So use of Qdos for that set led to a 23% increase in full text downloads on the publisher site. Now that was a really, really interesting finding for us.
15:41
That's been written up in PLOS One, and I'll include the link on the slides there so you can have a look at the full article there if you want to. So opportunity, attention, effectiveness, we map that to reach resonance and results. And it's the reach area that I think is the most interesting because that is the start of the whole process. And if we map this to fishing,
16:01
I promised Hans I would talk about fishing, and showing a nice sunset after you're all a bit sleepy after lunch is a bit risky, but I'll do it anyway. So if we think about reach, reach is asking the question, where is the best place to fish? Where are the best pools to fish in? Where am I most likely to get the results I want to? Resonance is how do I get the fish to notice me?
16:22
What technique should I use? What bait should I use? What's the best way to actually go about this? And results is how many fish did I catch? So that's quite a nice sort of simple way of thinking about reach resonance and results in this particular model. So last slide, what happens next for us?
16:40
So we're very much about broadening the model. So we want to include additional data sets. We're gonna be talking to a number of additional organizations beyond the ones we're working with now to have more indicators for both opportunity and for effectiveness as well as attention. Building that bigger data set, the algorithms to interpret all that data, the intelligent recommendations that come from that.
17:02
And we're really interested to hear from institutions, perhaps institutions in the room today who are interested in this and would like to perhaps learn a little bit more about how they might benefit from this. So we're sort of at the very early stages of playing around with data sets, but I'll be here for the next two days. If you want to sort of talk about that, then please do come and find me. And on that, I will leave you, and thank you very much indeed.
17:21
Thank you. Thank you very much, David. Kudos for David. There's time for one or two questions now. And if there are questions after this session, we may have time for that too. Any question from the audience? And if people coming from the social media lab tour,
17:43
please move to the front. Don't hide there in the background. You're welcome here in front to sit with us. Any questions? It was clear? Let's keep your questions for the end of the session.
18:00
Next to our, our next speaker will be Sasha Nuchowicz. He's the editor-in-chief and founder of STEM Fellowship Journal. And he's working a lot with students. So that's gonna be a really nice, interesting new angle to the matter. The title of his talk will be Using Examples of High Altmetric Scores
18:21
as a Roadmap for Pre- and Post-Publication Editorial Support. Please welcome Sasha.
18:47
I will hide behind the podium.
19:06
I would like to address in this presentation three points that I hope will be of interest to the audience. Firstly, it will be two distinct types of scholarly publication. Second, I think that I can define
19:25
the main pillars of scholarly media marketing. Some form of, let's say, beginners' scholarly marketing guidebook. And then finally, I believe it is really a big thing
19:42
on my last slide. Okay, so to ensure scholarly publishers' success today, it's important to understand the ever-growing professional audience online and conversation happening on the networks.
20:01
Altmetric score tabulates the online reach and how it was asked already. It is not only positive reflections. It is all reflections. And in nature, the publisher should focus their social network activity to gain additional traction
20:22
following the patents of Altmetric. The Altmetric score captures the online and social network footprint of a paper. Its analysis and composition sheds light on two distinct forms and expectation
20:43
when it comes to academic publishing. One is actually something which is traditional and we understand that when it comes to scholarly publication in common sense, we basically need to alert professional community
21:02
through the social media. And it is a way of sharing a reputable opinion through the social media. What I want to point your attention to, it is actually another new class of scholarly publications. It is papers where feedback from online community
21:24
is much more significant. Academic quality control here has been drifting from publishers actually into public domain. I don't know if you personally experienced or dealt with this type of papers.
21:42
I specifically want to point you to computational science papers which value social media opinion much higher than reviews of editorial boards. For this type of computational biology,
22:04
computational physics, AI, it's simply the opinion of the conference is absolutely paramount and it is totally different nature of research.
22:20
This is totally different nature of science where any experiment can be repeated by nearly wide audience of the listeners or participants of the conference. That's a critical difference between these two classes of research
22:40
and scholarly publications. In both cases, altmetric score is a roadmap to follow in gaining online attention to research publications. But again, when we're looking at altmetric score, it is 15 platforms which they are following to define. When we're looking at the top altmetric papers,
23:03
top scores in altmetric, you will see that they are defined by maximum seven, eight of these platforms. So then I would like to suggest to look at the,
23:20
to look at the free publication and post-publication responsibilities of the publisher and author if they want to achieve success on the social media. I would like to point to you and draw your attention to this graph.
23:40
This is a digital news life cycle from 2020 social. And as you see, this graph is actually built on SMS alerts, blogs, content analysis.
24:03
So in case of scholarly publication, you can actually have pillars to support this graph to actually curve it around by following, by creating your own tweets, by creating hashtags,
24:21
editorial blogs, Google Plus posts, which were already mentioned here as inefficient, Wikipedia references, ideally articles, Reddit posts. Scholarly publishers and editors need to engage in digital media marketing beyond brand promotion.
24:44
Their goal should be leadership in the online expert community. And I was glad to hear from our tour that actually the lab is researching the role of Reddit moderators. And this is the role a scholarly,
25:03
the role editors should be taking on, moderators on Reddit. And of course, be responsible for their own blogs. I would like again to quote the same source as I mentioned, 2020 social,
25:22
and point to you the power of group forming. So here we see the first chart. It is a broadcasting network where 400 nodes, 400 people, you're establishing 100 connections.
25:40
Then we have a telecom network where 400 nodes, 400, and users, we having and to the power of two connections. And finally, we have something what is actually was demonstrated by what CUDA does, creating actually to the,
26:06
creating two to the power of N connections. Okay, actually it was already mentioned. Network active editors with a strong professional
26:22
online presence give a competitive advantage. And where they should be, they should be on Reddit, they should be on Quora, they should be on Wikipedia, establishing and boosting altmetric scores or any other score regardless of counting system.
26:45
This is simply marketing procedure. There is also a new role of students who are mostly excluded from editorial process. You can consider using bots mentioned
27:03
in the keynote speech today, or you can actually go much more efficiently by engaging students on Twitter and Facebook and populating and supporting publication with a student post.
27:20
And now here is my final slide. Oh, not final. No, here is my final slide. I would like to draw your attention to the picture. I worked hard on it.
27:41
Well, at least I put a lot of thought. So basically, what we're dealing here with and what I want to spend my last slide on doesn't relate to the topic of my presentation. I want to tell you that social network footprint
28:03
of research and what you are having in your hands is a calendar. It is the golden mine. You simply don't see the value of yet. It is simply a new type of learning resource.
28:20
What Kuda was mentioning and what Kuda already exploit to get this high exposure, it is actually new way of people discovering information online. And this is old people discovering when we deal with modern students, they are digital learners.
28:41
And when we talk about digital footprint of scholarly publication, everything what Altmetric shows from news stories, YouTubes, Twitter, Facebook, it is actually a learning field
29:02
from which students can actually feed and acquire the concept of research by going in their own way in their own learning patterns. Different from traditional pedagogy, different from university style of delivery of materials,
29:24
but with efficiency much higher. So it is actually a way for you either to become a curriculum delivery institute, I'm talking about publishers, or a way for publishers and libraries to become immediate partners of universities,
29:41
providing quality self-directed courses based on the footprint of subject specific issues or volumes because any issue on one subject, it is already ready to use course. Thank you.
30:04
Thank you, Sasha. Any burning question right now? Not yet, right? We're getting back from lunch. Everybody's in now. Our next speaker will be Camilla Lindelöf from the Södertorn University in Stockholm, Sweden.
30:22
She will talk about can Altmetric data help researchers fine tune their publication strategies? And I believe she's using Lamex data for this. Please welcome Camilla.
30:59
Yes, so Södertorn University,
31:02
it's a small university, very small university compared to your parts here. But still, the university is talking a lot about publication strategies. And currently there is no consensus of publication strategies. And I will just give some examples
31:21
since we are in the, this is a work in progress of what we are doing and my thoughts as being part of this research support unit at the library and doing outreach seminars with researchers in my work. So this is what I'm going to present.
31:42
So I have three issues I want to raise. And the first one is that I want to talk about Altmetric data as a discussion source, a source for discussions with the researchers, one discussion source. And also, I want to talk about the individual perspective
32:02
versus the group perspective that I find when I talk to researchers seem to be important. And also, a thing about chasing shadows, we will get back to what that means. And I understand I have challenged you here
32:20
with my slides because the next slide will be challenging. So I put the, I did like the others, and put my bit.ly link over there. And put the slides in my institutional repository. We'll see how that will make up. Okay. So from this graph, here is a graph
32:42
where I've combined data from our institutional repository and Plamex data. And in this example, you see a lot of dots with different colors. Every dot represent a dissertation written by a scholar from working at Silicon University.
33:02
Not written at Silicon University. And there's a timeline. The x-axis is the publication date of the dissertation. And the y-axis is the WorldCat count. The library holdings count. We can discuss another day if this is an Altmetric data.
33:23
But, yeah. And the colors of the dots are languages of the dissertations. What language is the dissertation written in. And I will come back to why it shows this particular indicator in my example.
33:42
I wrote a blog post for this conference before and I used Facebook data instead there. And so I just noticed that this is 40% percent of our dissertations showing up here. 60%. 40% is falling out of this analysis.
34:07
And so I haven't looked further into that but it's probably a difference between comprehensive dissertations and monographs, for example. And yes, I could add that in Sweden we write dissertations
34:22
and they are published as a final work. I think here in North America you don't really publish your dissertations. You rewrite it later on. Correct me if I'm wrong, yeah. And so people are interested in their dissertations and how they spread. And one thing that I just want to note
34:42
is that the highest points, the outliers, is about 125 library holdings, just to note that. So in my example here, I have a seminar coming up now this autumn where I will talk to PhD students about their dissertation choices,
35:02
a kind of publication strategy. They have to decide whether to write a monograph or a comprehensive one. And I have a quote here translated from Swedish. There is a PhD student saying, I will definitely not write a monograph in Swedish. It will ruin my future research possibilities.
35:21
He was very clear about this. And he's a PhD in educational studies. And so from the colors of the diagram, you could start a discussion and say, yeah, you see that the English written ones are spreading a little bit more.
35:40
We don't know if it's Swedish libraries or if it's world libraries. I would say that there's not such a big difference between the Swedish and the English written ones. I don't know if you agree with me or no, if you can see the colors at all, but it's quite a spread. You see that they have to have some years
36:01
to spread in libraries. Well, and you can also ask the question, if this is an indicator I would use in this case, maybe along other indicators. But yes, when I talk to researchers, they are in humanities, particularly, they are interested in library holdings still,
36:22
and they are interested in printed books still. So they see this as an interesting measurement. Another PhD student I had the other month, he told me, my supervisor will probably forbid me
36:43
to write in English. So you can see the outliers, I lined them in there. These outliers are six dissertations and they are all monographs written in English and by now senior scholars.
37:00
These are written at the bigger universities in Sweden, Uppsala, Stockholm and Lund, well known, even outside Sweden. And they have some years now when the last one was published in 2004. But so here I just want to point out being a new PhD,
37:21
this one is a new PhD, and relying on a supervisor. And I think this is a thing about supervisor power. When you have to deal with your supervisor and they will probably give you very good advices, but it's one person. And in a seminar like this, you can come to discuss is this a good advice to follow.
37:42
Probably this supervisor is not one of the supervisors who wrote the dissertation in English. I don't know. But I quote from nature here about open access journals and it's the same kind of idea that a PhD student have to discuss with a supervisor and the supervisor has kind of the last word
38:03
about publication decisions. When I talk about a group thing, I'm thinking about researchers' response when I meet them and talk to them. First of all, I just want to point out
38:22
that the validity of individual metrics are these valid for others and the researchers themselves. It's a common question in bibliometrics. We can discuss that. What I found when I started to do this seminar is I asked individual researchers, can I use your data?
38:44
And yes, I could, but they were afraid to come across as not being good enough showing their altmetric score. And other researchers said, yeah, this is interesting for me,
39:00
but I don't want to be judged or evaluated by this. So common in humanities, for example. So I thought, what if I start to do this on a group level, showing the data on a group level? And with more data, you can also look for patterns. I mean, now we are in a very small,
39:20
we are within our publications at our university, so you cannot generalize and say this is true for everyone. But still, it's kind of an invitation to explore data together and as a discussion source. So the third point about trying to avoid the shadows, in search for the substance. I borrowed this sentence from this review
39:42
that was mentioned earlier today by Sugimoto et al. And some suggestions I also found while working with the researchers is to emphasize that this is an exploration phase. We don't have the answers with this data.
40:04
So we are exploring the data together. And the heterogeneity, well, we had talked today about societal impact, and that is some of the indicators. But for example, the example I showed here is about library holies.
40:21
It's not probably about societal impact. And so be clear about the different target groups. Data sources, it's really interesting because we usually end up discussing research gauge, for example. And so it's a good discussion. How do you deal with, as a researcher,
40:40
being part of these platforms? And clearly visible and less visible biases. Yes, the less visible biases, like the DOIs, I have some troubles with them because it's not clear for publication types that the DOIs, we rely heavily on the DOIs.
41:02
So I have to make a point of it several times. And also, just one thing more, is that I like to start with the discussion from their point of view, because I used to start with defining altmetrics,
41:22
bibliometrics, and so forth. And I don't find this a fruitful start with the researchers because they are not interested in if altmetrics is like that from bibliometrics. They want to know what can the data tell them about this. So, finally, can altmetric data help researchers
41:42
fine-tune their publication studies? Well, I do think so, but we have few straight answers. We have to explore the data together. And I think the library can help the researchers doing this by having these seminars, or we are invited to talk to them.
42:01
But it's a discussion thing. And it's a testing and development phase. And yeah, I mentioned this about fruitful discussions. And one final and important point is that not every researcher will find the same way.
42:22
For example, the PhD students that I mentioned before, one will probably continue to write in Swedish, but hopefully he has the arguments to talk to his supervisor and make an informed choice about this. The other one will probably continue writing in English because he was very determined to do so.
42:41
But he will also have helped in the discussion within the seminar. So, thanks a lot for listening. Thank you, Camilla.
43:00
I think there are quite a lot of librarians around here. Do people recognize this story? I see some hands going up there. Do people recognize it, or is it totally different from their particular situation? Anybody? Martin?
43:23
Oh, Martin. Martin, I was thinking you were asking a question, but he's running away. Okay, next speaker is a duo. Two ladies from the University of Guelph. I'm gonna introduce you to Melanie Cassidy
43:42
and Ali Slaus. It sounds like a very Dutch name, but she's also from Canada, so they're both from the University of Guelph. And the title of their talk will be Who's Talking About You? Using Optmetrics in Library Assessment. Please welcome Ali and Melanie.
44:15
No, I know what to do. I'm good.
44:26
Hope everybody enjoys watching me fumble with the computer. You can't even see it on there. Yeah, I know, I know. I'm good, don't worry. It's okay, Claire.
44:42
Hey, everybody. Thank you for having us. I'm Melanie. This is Ali. This is our presentation. It's gonna be very exciting. There's a copy of our presentation at bit.ly slash 4AM underscore assessment, and then there's gonna be a copy made available to you through the usual channels, I guess,
45:01
after the conference. So yeah, this is who we are. We just wanna take a moment to acknowledge our co-collaborator, Melanie Parlett-Stewart, who could not be here with us today. Oh, that sounded ominous. She just couldn't make it. She's totally fine. So these are the things we're gonna talk about today.
45:22
Gonna give you a little bit of context of what this actually is about and the process that we used, and then a couple of things we wanted to touch on a little bit more deeply. It's a lot to get through, so I'm gonna go through it kinda briskly, but don't be afraid to come and talk to us afterwards at coffee if you have any questions or you can shoot us an email.
45:42
So we decided we wanted to try using altmetrics to assess a library program that's called Take a Pause. Take a Pause is a pretty standard event on many library or university campuses. Therapy dogs are invited to the campus and students make appointments to meet the dogs and pet them and play with them. It's really magical.
46:01
From our traditional metrics, which are things like how many people were signing up and how many people were attending the event, we knew that Take a Pause was pretty successful, but we thought it might be interesting to also see what people were saying when they were in their own spaces. So effectively, the core of our project was about meeting our users where they were and seeing if we could learn something
46:22
about their habits, their interests, and their engagement with Take a Pause. This is the process that we set up for the project. The first step is what to assess, and I already kinda covered that. We're assessing Take a Pause. For you, it could be a specific event, a specific service, or you could be looking at the general attitude of your audience
46:41
or your user base or a subsection of your user base. So again, we're in an academic environment, so maybe we wanna see what graduate students think of us. This is something we could apply to that audience. Step two is looking at the many ways in which our service might be identified. So for the librarians in the room, this is a lot like brainstorming a list of keywords
47:01
before you do an actual search, effectively looking at what language our users are likely to use when they're talking about Take a Pause. Hand in hand with considering the many ways in which people might identify our service is considering where to look. Ali's gonna talk about this a little bit more in a minute when we cover the things to consider.
47:21
Following that are steps four and five, which might seem like they're the most obvious steps, but the process of collecting the data will likely change depending on where you choose to look. For example, some of the social media platforms that we considered made data collection either impossible or very difficult. And where you choose to collect the data
47:40
can of course have a big impact on how you're able to analyze it. So you might wanna consider step five even before you approach step four. Okay, I'm gonna turn it over to Ali. She's gonna discuss the last couple of steps as part of the things to consider. Let's switch places. All righty, so really underpinning all of this
48:03
is two core considerations. So the behavior of your users and the purpose of collecting the data. For example, like Melanie said, are you trying to evaluate the success of an event like we were? Are you trying to test kind of the development of a new service model? Do you think that your users are familiar with Twitter, Facebook, other social media tools?
48:22
If so, do you think that they're gonna post about the event or the service model on these platforms? So these two things are really important to think about because they're gonna shape the assessment process. So it's really kind of important to give lots of time to that consideration even before the data collection gets underway. So Snapchat is a good example of user behavior.
48:42
So there's lots of user data that shows that Snapchat is a really popular platform for university-age students, but it doesn't kind of keep the data in any readily available format. So it's not a great option in terms of mining and trying to analyze after the fact.
49:02
Another thing that gets overlooked in many cases is the preservation and sharing of data. So while you've done the legwork and you've really figured out what you wanna use the data for, you might wanna consider whether this data will be of benefit to anyone else in the library or anyone else at your institution more generally. So this is where you wanna start thinking about
49:21
where you're gonna keep your data and who will be able to access it. So we've listed a few options here, but really the options that you end up using will be dependent on your local resources and needs. And because what would an academic presentation be without some caveats,
49:40
we also wanted to acknowledge some of the limitations of this as well. So you might run into any number of barriers which could be either related to content, so maybe no one's talking about you, technical issues, so maybe the data is too hard to find or understand, or staffing. Tools do take time to learn, and again, depending on your resources,
50:01
you may find that this isn't something that you can kind of sustainably explore at this time. You may also find that the lack of data tells you something, or may kind of change your perspective. So if no one is talking about you, might there be a reason why that's the case? The idea of using altmetrics for assessment in libraries
50:21
is really very experimental, and we would know because as librarians, we did our due diligence and ran a literature search on it first. But there are still a lot of kinks to iron out, but the best way to kind of do that, I think, is to kind of just explore this, to try it out, see what works or doesn't work for you,
50:40
and the questions you're trying to answer firsthand. But we do think that this is a worthwhile exercise, because you can usually glean a lot more information from folks when they are on social media or talking to people in their network that they trust, rather than people who are trying to collect data in a more kind of like formal way
51:03
through surveys and things like that. That's it. Any questions? Melanie, do you want to talk about that? Okay. The question was, what did we find? So to be totally honest, it's still a little bit ongoing,
51:21
still analyzing the data, but I can tell you one thing that we discovered is that students love take a pause, but because the spots fill up so quickly, they have a really hard time finding opportunities to go with their friends. So that's data that we can share
51:40
with the people that organize take a pause, and maybe there can be ways that we can accommodate group bookings or duo bookings so that people can actually go and play with the dogs with their friends, because they're a little disappointed when they can't do it. Any other questions?
52:08
Or we can repeat it in our mic too, whatever works. So what data sources did you end up using that you could get some useful analytics out of?
52:21
We looked at comments on Twitter, Facebook, and Instagram, and we also looked at the number of responses that original comments were getting, and what kind of reactions people were having. So like likes and favorites and retweets and stuff like that. But the bulk of it comes from analyzing
52:41
the free text comments. And did you look at whether they were positive or negative, or did you do analytics around that? Oh yeah, absolutely. We looked at whether they were positive or negative, yeah. Can I just make one more comment?
53:00
Okay, so this could have been a very long presentation. This is just kind of a taste of what the presentation is and if you wanna check out the bit.ly, you can see the presentation that we have, and there's also a workbook from another presentation that we did on this topic that might help you
53:21
to understand the process a little bit better. It's actually a workbook to go through the process, so you can go through every step in our process and figure out what works for you. Thank you Melanie and Ali.
53:41
I think we've seen some great examples coming along about these practical applications of all metrics. I do remember from yesterday, and I think it was Hermana Bata asking, what could kudos do in applying local languages in reaching a bigger audience?
54:01
That's basically what your question was, right? David, could you address that question? Yes, that one's for me. So one of the things that we do is we work with organizations that will translate, if English is your non-native language, to create plain language summaries in English. We find that people are actually using both languages,
54:23
so we have people writing plain language summaries in Arabic and English, or in Japanese and English, and that's increasing the search engine optimization for them, so we haven't done any hard experiments on the data yet to look at significance and so on, but it seems people are wanting to do that,
54:40
and as we get a higher volume, we'll certainly look at releasing those figures. But yeah, it makes sense if it's in multiple languages. Also, forcing you to write a plain language summary of English is not your native language actually helps you clarify your thinking about what actually it is about and why it's important. So there might be an effect there as well.
55:02
Okay, thank you very much. Any other further question before we wrap up for this session number three? Thank you very much, all speakers. I think we've learned a lot about weighing the pick, about fishing, about working with students, and how to deal with also language and psychology,
55:21
and I think that's also a nice reflection of what we discussed yesterday. Thank you very much for your attention, and I'd like to hand over to the next session, being chaired by Andrea Michalek on novel and experimental uses of altmetrics. Andrea, the floor is yours.
55:41
Thank you.
57:26
I love the creaky microphone. It really wakes everybody up, right? Short presenter coming up as it creaks down. My name's Andrea Michalek, and I was the co-founder of Plum Analytics, and about six months ago,
57:41
Plum Analytics became a part of Elsevier, and so now I'm both, I have kind of a really interesting role that I'm both vertically oriented on a product for Plum Analytics and the PlumX dashboard product, but even more excitingly, I kind of look horizontally across all of research metrics across Elsevier.
58:00
So in the past six months that we've been there, we've been doing a lot of work to take all of what we've done with PlumX metrics, integrate them in all different types of products, and make them even more available and freely available for people to interact with. So since I started Plum, one of the most enjoyable things is seeing what people actually do with the data, right?
58:21
We're an aggregator. We pull it all together. We try to pull the right stuff together, but we really love to see what people do, and so this session is all about innovative uses of altmetric data. So it's a great opportunity as we've kind of laid the groundwork so far in this conference to see some people kind of playing at the edges, at the fringes
58:40
of what you can do with this data, new ways to visualize it, new things to learn from it. So I'm really looking forward to three great presentations here, and first up is temporal visualization of altmetric data across heterogeneous sources. Introduce yourself.
59:01
Thank you. Hello, everyone. I'm Vikas Fauja from the Knowledge Discovery Unit at the National University of Ireland, Galway, and this is a work that we did as an industry-targeted project with Algebir. So we wanted to know that how does research disseminate from the formal publications
59:22
into real literature, and while we did all this work, we were continuously trying to agree on a definition of what real literature is. So first we thought that all the government documents would be real literature, government policy documents, white papers, and then eventually we agreed on
59:42
that pretty much everything which is not research paper would be real literature. So we wanted to know that how does the research impact, somehow, if we could find out, that how does the research impact.
01:00:00
flow from the formal publications, general papers, conferences into these alternative sources that we were vaguely calling as great literature as well, which were the white papers, government policies, blogs, news, all those items. And would we be able to detect such dissemination of resource or citations in the informal
01:00:25
literature, heterogeneous sources? So we agreed on two keywords. We agreed on the keywords of Tamiflu and HPV vaccine, as discussed in our project. And we wrote a little crawler that went over to many government websites, a list that we
01:00:44
collected, World Health Organization, CDC, UK health related websites. And then it crawled documents, a search for documents containing these keywords on those websites, and it downloaded them. And we started a manual analysis of that.
01:01:03
Now, we found out that although these documents contain a lot of information that is hopefully coming from scientific sources, like for example, during those days, there was an official NHS policy that the dosage of HPV vaccine would now be two doses
01:01:22
instead of three doses. Or we were looking at some US legislation where they would say that the age for the dosage would be this, this, and so you would get your first dosage at 14 years instead of this year. So all this information presumably backed by scientific research, but there was no mention
01:01:42
of where it was coming from, who was the scientist, what was the research publications. But we did find out by our manual analysis that there were a lot of mentions of scientists and research organizations in the form that scientists from this university said that or they reported or they told.
01:02:01
So we formed this as our basis and we did some basic annotation in NLP and we used a tool and tried experimenting with it using these sort of trigger keywords and putting them in different combinations to see if we can find mentions of scientists or research organizations.
01:02:21
And with our experimentation, we were able to develop some patterns. For example, now when we annotated that, the small corpus that we have, it detected all the person, well, most of the person names within the corpus and most of the organization's names within those documents.
01:02:40
And then we developed patterns like we developed trigger words like a person name followed by any other word and then told and then similarly according to reported, we had trigger words like these. So if you can see here that the academic organization detected in the text is highlighted
01:03:00
by green and similarly a trigger phrase is highlighted by blue. Another thing that we did was since a lot of person names appeared, so we had access to the Scopus database at that time, so any organization or scientist name appearing in our documents, we would go to Scopus to make sure that this was a scientific name.
01:03:21
Now we hardened our patterns and then we converted that into a formal sort of NLP pipeline and we had access to almost 12 terabytes of data collected between November 2010 and July 2011. This is the window. We search this data for our keywords and then we put all this data through the NLP
01:03:42
rules that we made. And then based on the, we received all those annotations in our documents. I would like to show you an example. Before that, a colleague of mine took that data and he did some graph metrics based on that in which he showed that H index wouldn't relate to graph measures like car
01:04:02
centrality or page rank. His work has been accepted in the Social Network Analysis and Mining Journal and it is under a minor review. Now, I do not know much about the subject, but I can provide links to his research if anyone is, okay, I'll do. Now coming back to our topic, so if we pick up a scientist, Gary Nabel, and if we
01:04:24
look at their Scopus profile, we would see that they have 454 publications and they have around 39,000 citations. Their H index is up at 106 and these are the co-authors that they have been working with. Now looking at our system, we also know that they have been co-mentioned 19 times
01:04:45
with the Center for Disease Control and Prevention and the number of co-mentions with other organizations as well. We also see that what were the sources of these documents where they were co-mentioned with. For example, 30 of them were news items, 21 were blogs, and 17 were government
01:05:03
documents, probably government policy documents or announcements, things like that. And this is, again, this is only between the short window of 2010 and 2011. Now we also visualize that in a force-directed graph and in this you can see that all these squares are documents, the circles are organizations, and diamonds are persons.
01:05:25
And then the color of the square indicates that what is the source of this document. So if you can see on the left side that there is a green square linking a lot of diamonds with this scientist. So with the green color we know that this is a blog document and it has all the
01:05:45
other persons mentioned within that document, including the scientist that we are looking for. Similarly, down below if you see there is a pink square which is a news item which is linking this person to all the organizations which are in orange color.
01:06:00
And we can also see if we want to, the content of the document, if we click on the document we can see that what the document is about, all the annotations are highlighted and if you, I don't know if you're able to read or not, but the yellow one is a direct quote that this person from this university is saying this. Now we put this also in an entity link diagram.
01:06:23
This is just to show you that if you want to further analyze the relationship of a person with an organization, so if you click on them it would only highlight those specific organizations that this person is mentioned with and then the documents are highlighted as well and again you can click on the document to see the document content.
01:06:41
So then the next thing was that we wanted to see if they have any relationship with time. So what we did was that we plotted all the co-mentions of the person according to time. On the x-axis you can see that from November to July these are all the months that we have the data for and then we have the number of mentions.
01:07:03
And by looking at this we can see that the highlighted one, National Institute of Health, that the co-mentions with National Institute of Health have sort of gradually increased from November to July. And then the co-mentions with the John Hopkins University for instance, they only started
01:07:21
in March. And so this is another thing that we were able to see with the data that we have. Now another thing which is work in progress and we think that our data can answer is that we want to see that if there is a pattern of these co-mention appearances in
01:07:41
the literature, for example, do they start usually appearing first in blogs and then they are picked by news documents or it's vice versa. First the mentions of a scientist or an organization start appearing in news and then they are picked up in blogs. So we're planning around this is still work in progress on a visualization like this
01:08:01
where on the x-axis again we will plot the time on the y-axis, we'll have the number of mentions and each line would represent the number of mentions from one particular source. So in this way we would be able to answer this question as well. Thank you for your time. Thanks very much. Hold questions till the end.
01:08:26
That's okay. Okay, brilliant. Yeah, we'll do it that way and that way maybe we'll get some nice interplay on all these different innovative ideas. So next up we're going to be talking, let me switch the presentation. That's an important part, isn't it? You don't want to see the same one again.
01:09:11
There's a joke here somewhere. How many people does it take to, there we go. Hi, good evening. My name is Arvind and I'm from Singapore.
01:09:23
I represent the art metrics team, which is working at the Center of Healthy and Sustainable Cities Research Center from the V. Kimbe School of Communication and Information. So I'll be presenting about this tool called Asaria, which is a short name for art metrics for research impact actuation.
01:09:43
So why did we actually develop this tool called Asaria? So this particular tool was built to measure the research impact of researchers at different levels, at the school, college, and also at the university level. So we built this tool specifically for our university, which helps in comparing the
01:10:02
performance of research from different disciplines. So what are the data sources that we use in this particular system? So we use both academic, that is bibliometric sources and social media sources. So we use Scopus mainly for getting all the metadata of publications and the citation
01:10:24
count, and we also use web of science and cross ref, along with Google Scholar. And for the social media sources, we use mainly artmetric.com data and plumix.com data. So in terms of the data extraction methodology that we use, so we use the Scopus API
01:10:44
and we use the affiliation ID of Scopus to retrieve all the publications from NTU. So once the data is retrieved, we use the DOIs from the Scopus publication. To retrieve the author names from cross ref, because Scopus API doesn't provide the full
01:11:02
list of author names, so we use cross ref to retrieve all the author names. So after this, we retrieve the citations data from web of science, and we use the Google Scholar profiles for extracting the citation count of publications. And then we use the artmetric fetch API for retrieving the artmetric data from artmetric.com, and we also use the plumix public API for retrieving the
01:11:26
artmetric data from plumix. So this particular, as I said earlier, this particular system is mainly useful for a particular university. So this phase one, we have developed it for NTU alone. So as of January 2017, NTU has around 4,000 art researchers, excluding the PhD
01:11:44
students and gen faculty. So we managed to retrieve the publications of around 2.6k researchers. This stands for around 86,000 publications, and we also only consider the publications which have DOIs.
01:12:02
Then since we collect all the publications from Scopus, we have an in-house author named disanguition algorithm to retrieve only the researchers that we want. So what are the different metadata fields and metrics that we have in the system? So all the basic metadata that we have in Scopus is available in this, and we have
01:12:25
bibliometric data and also the artmetric data. So at this point, we are considering only the tweets, views, downloads, and mandatory reader count apart from the artmetric attention score, because we have ignored all the other metrics which are suffering from data sparsity. So currently, we just have these metrics in the system.
01:12:43
And the frequency of the system is all the way from monthly to yearly level. Since this particular system can be viewed at an aggregated level, we also provide aggregation at school, research center level, and also the college level. So you could see the aggregated data for top five, bottom five, or, you know, 15,
01:13:04
20% of the school. So in this particular system, we have two profiles. One is the researcher profile, which is for a particular researcher. So you could see your own data in all the visualizations, and then you have another profile called as the admin profile. So this particular admin profile is useful for heads, chairs, or even dean of colleges
01:13:26
and schools to view the data at an aggregated level. So we look at some screenshots of this particular system. So this is the dashboard screen. So our principal investigator is a big fan of the tile design in Windows 10 operating
01:13:42
systems. So she wanted to have these different tiles representing the different aspects. So you have bibliometrics, altmetrics, then you have the artifacts, which are the publications, and then you have the cross-metric explorer and map view. So at the bottom screen that you see for the admin dashboard, the only difference
01:14:00
is it doesn't contain a map view, and it has an additional button for selecting the entities. So what is the entity selection about in the admin dashboard? So as I said, the admin dashboard is specifically useful for viewing the data at an aggregated level. So you could select the school, colleges, or research center, whichever you want. So this is specifically for NTU.
01:14:23
So we look into the first visualization, which is the publication and citation trend. So this comes under bibliometrics. So this particular visualization basically visualizes the number of publications versus the QS and non-QS citation. Okay, I'll tell you about QS citation. So QS citation is a new citation that we came up with based on the affiliation
01:14:44
of citations. So normal citation, you would not have any prestige element to it. So we wanted to find out only those citations, which are from top universities. So we use the QS world rankings to identify the top 200 universities. And with that, we use the citations affiliation data to identify whether those
01:15:03
affiliation are from the top 200 universities, and that was classified as a QS citation. So if you have more QS citation, it means that your paper is cited by top researchers. So that's what we display here. So there are two blocks. The first block is the number of publication.
01:15:21
The second block is the QS citation and non-QS citation. So at the bottom, we have the data available in table format. And also at the bottom table, we have the statistics like mean, median, average, correlation between the publication count and the citation count. So the same visualization is available at the aggregation level.
01:15:41
That is for the admin dashboard. So the user can click on any of these bars, and the user can see who are the top 10 researchers for that particular year which he clicked on. So he could see the bars for both citation and publication count. So this way you know how your researchers are performing for that particular period of time.
01:16:03
The next visualization is bibliometric time series. So this particular visualization is similar to the previous visualization, but this is more at the quarter level. You have the same metrics displayed, the publication count and the citation count. So when the user clicks on any of the bars, it displays another line chart in the drill down.
01:16:24
So you could see the top 10 researchers for that particular quarter. Then we move on to altmetrics section. So in this visualization, we compare the publications against the altmetric score. So there are two bars. So the first bar basically says how many articles are with altmetric data
01:16:43
and how many articles are without any altmetric data. So you know which of your articles are there having altmetric data. And the second bar basically represents the cumulative altmetric attention score for that particular year. So similarly, the user can click on any of these bars and see who are the top 10 researchers for that particular year based on the altmetric data.
01:17:07
So this is based on the altmetric attention score data that we get from altmetric.com. And then the next visualization is called as altmetric time series. So this particular visualization is specifically for tweet counts, visualizing the tweet count.
01:17:21
So you have the year in the x-axis and you have the count in the y-axis. This is a pretty straightforward visualization. And again, in the drill down view, you would be able to see the top 10 researchers for that particular month you have selected. So this particular visualization has a scale bar at the bottom. So you could just move it and it would show you the months in a particular year.
01:17:44
Now moving on to the artifacts section. So artifacts section is basically just for viewing the data for publications, journal papers and conference papers and also books. So this is how the page will look. So basically we just list down all the publications for the researcher
01:18:01
and we also list down the metrics at the bottom of each publication. The one interesting thing about this particular page is we also show the journal impact factor quartiles for that particular journal. So you could know whether it's in the top 25 quartile or 50 quartile, that additional information that you can see here.
01:18:21
And also you can click on any of the citations and see the list of citations for that particular paper. So this is how it looks for the admin dashboard. So in the admin dashboard, as I said earlier, this can be used for viewing data at an aggregated level. So we provide an option called as tree search, which could be used to select the researcher you want,
01:18:42
because a particular school or a college might comprise of many researchers. So we use this particular visualization so that you could click on the school name, then you can click on the faculty or the research section and you can click on the different role and select the researchers. So this is useful for easier navigation.
01:19:02
So we have the same page for books. And then finally we have the cross metric explorer. So cross metric explorer is a kind of visualization which will allow you to select any metrics that you want. So previous visualizations that we saw, all the metrics were fixed. So in this particular visualization,
01:19:21
you could select any two visualizations that you want. You could go to the dropdown here, sorry. You could go to the dropdown here and you could go to the dropdown at the top of this page and select the metrics, select the time period and then select the frequency and view the data.
01:19:41
So you could select any two metrics. And this particular dashboard for the admin user has a lot more features. So for example, this particular screenshot that you see is for a particular school. So the user has selected a particular school and for that particular school is seeing the scope of citations and the Web of Science citations
01:20:02
and the aggregation type has been selected as medium. So what basically this means is of all the researchers in the particular school, the median value is calculated and is visualized. So you can see how the data is at different aggregation type. So in the aggregation type dropdown box, we have median top five percentage, top 10 percentage, bottom five, bottom tens.
01:20:25
So basically if you select top five, it would take all the top five researchers from that particular school and calculate the average. So this particular use case is for the school and then this particular use case is more useful for a dean of a college
01:20:41
who wants to compare the performance of two schools so you select the two schools and then similarly you select the aggregation type, then you select the metrics and you can visualize the data. So this helps you in comparing the performance of different schools in a particular visualization. And the third one is the interesting one where you could compare a researcher against a school.
01:21:02
So you can select the researcher in the top right-hand corner dropdown box of a particular school and then you can select the aggregation type. So what's being done here is top 20 percentage aggregation type has been selected. So we calculate the average of the top 20 percentage of researchers in the particular school
01:21:22
and compare the data with this particular researcher. So you could see how we are doing with the top 10 or top 20 percentage. So this visualization could be used for checking the performance of researchers while looking for promotion and all those things. And then finally, this is the map view.
01:21:41
So map view is a visualization which is for using the affiliation of the citations. So this particular map displays all the countries where your citations are from. So you have the different legends. So the different legends represent the percentile. So if most of your publications are from a particular country,
01:22:03
that particular country would be among the top colored country in this map. So this particular visualization helps you in seeing which are the countries which cite you the most often. And you could select the year and the quarter of this particular visualization and see accordingly.
01:22:21
So that's it about the visualizations. So I'm just going to use the last two minutes for some shameless publicity. So we are conducting one workshop next year in our university. So it's called as Arazim. And this particular workshop, the papers will be published as a part of Springer CCIA series.
01:22:41
So we have two keynote speakers, Mike Delval and Stacey from altmetric.com. So the submission deadline is basically next month, 10th. So please do consider this workshop. Singapore is a very good place to tour. Okay, thank you. Oh, too advanced.
01:23:31
Great. Last up here is using altmetrics to highlight academic research, some of the innovative possibilities from York University.
01:23:40
So that's strong here, right? Thank you. Good afternoon, everybody. Thank you very much for your time. Thank you for being here. York University is located a little further north from here. There will be a direct train from the subway station here, which will take you to York University starting December.
01:24:02
As I said, it is in Toronto. Now, I will be talking more about the free tools that you could be using in your university and doing something similar to what I have done over here. Basically, I was using PubMed database from the National Library of Medicine,
01:24:22
doing an affiliation search for York University, trying to find out the papers which have the highest altmetric attention score. How was I doing that? I was running that affiliation search in that free altmetric for librarians, a tool that if you all know about, great.
01:24:41
Otherwise, there are altmetric guys over here who will tell you about that. So you would run a search in Altmetric Explorer. You would get the top 100 papers with a high altmetric attention score. I would find the unique professors from York University, and I was writing to them and telling them that this is your score.
01:25:03
I presume or I hope you're happy. Give me three minutes of your time and complete a research metric survey. So some of those things I'll cover here. And finally, there are three or four blue skies slides for the altmetric guys and for you guys as well. So York University has this online newsletter by file
01:25:23
where some of the researchers and their work is mentioned. But not everybody is mentioned over there because of space constraints. And it's not very clear how exactly some work gets mentioned over here. But when I was looking at this work by this prof published in Lancet,
01:25:42
I also saw that he had a very good altmetric attention score. So I wrote to him, listen, this is your score. And he was very, very happy. And he was talking to his researchers. This is Rajeeva sent this score. And he's asked us to complete a survey. And we should complete a survey. So I said, why don't I try this with all the faculty members at York
01:26:03
with a high altmetric attention score? So this is what I was doing. I was, this is the altmetric explorer. And you have over here the same search strategy that I was using down here. It'll be a little difficult to see. But I have that up in red, whatever search that I was running.
01:26:26
And you get all the top 100 papers. I was also adding in the Google Scholar citations. And I was downloading all this data into Excel and then transferring this into Tableau for data visualization.
01:26:41
So all that Excel chart goes here. And I was also adding whether those top 100 articles are the open access, the red is open access, excuse me. The green is, it's open on the publisher's website. If it is purple, it is on ResearchGate. So I was actually looking at all those papers. And this color sort of changes over the years.
01:27:00
The x-axis is the journals. The y-axis is the altmetric score, Google Scholar citations, and the news media outlets. So those are the quantitative measures. I was going from 2016, 15, up until 2012. And I'm looking at 2017 data. So you can get all that information from that graph.
01:27:21
And I constantly update it. Going by today's talk, I have seen that most of the papers, 2014 and 2015, are open on the publisher's website as well as ResearchGate, followed by PMC, PubMed Central, and then York Repository, the institutional repository.
01:27:40
So the color signifies that. I was also adding the subjects, what I could gather from the PubMed database, those top 100 papers. And I was just dragging that data and throwing it into Tableau, data visualization. And it just grabs that, okay,
01:28:02
he has said that this is about diabetes or obesity, and this is about honeybees, and this is about climate change. So you can come to know the top areas within your university. You just throw that into Tableau as long as you can assign certain subject headings to those PubMed papers, and then throw it into yours.
01:28:21
So you are more aware of the research that is going on in your university. And this is, again, the Excel data into Tableau. It has helped me connect with faculty members. I'm more aware of what research is going on. I've had some fantastic feedback on Altmetrics and research metrics in general. I was telling them, were you aware of your score?
01:28:43
This is, I have done a very small study for York, 37 individuals, and then for Ryerson, very informal. I've got eight responses thus far. So 26 of the 37 were not aware of the score. And so again, there was conversation on that.
01:29:02
I was also, I'm not going through all the questions over here. This presentation in much more detail was presented at ACRL in Baltimore this year. So the slides are more over there. But a few of the things which I did ask them was, how do you promote your research, your research papers? Again, I didn't ask about ResearchGate at York,
01:29:22
but in spite of that, in the open comments, they said, 10 of them said ResearchGate, and of course, Google Scholar Profile and the university webpage. Those are the main one. At Ryerson, I specifically asked, do you promote it on ResearchGate? That was one of the options, six out of the seven who completed said, yes, ResearchGate, we do promote it.
01:29:42
So I could see that. Had I asked it at York, I'm pretty sure that number would have been much more higher. After knowing about the score, I asked them, what is important for you guys? And they said that the number of downloads mentioned in news media, total citation count, of course, mentioned in policy document, these are important to very important.
01:30:02
Again, small slide, little difficult to read, but once you get the slides, please have a look at that and any questions, I'll be happy to answer. What metrics do you now want to know about? We want to know about the number of mentions in popular media. We want to know more about altmetrics and altmetric attention score.
01:30:22
This is, again, it was a broad scope of audience. The science people were definitely into the score and numerics and quantitative, whereas history said that we are not enamored by this, so no, and the no was here.
01:30:41
When I asked them, how important do you think is altmetrics for grant applications and tenure and promotion? So you can see the red is not important and the rest two are very important to important. Younger scientists who have just started and people in the sciences were definitely more for this,
01:31:02
that yes, it does play a role. We've been asked for these metrics when we are applying to grant funding agencies, private grant funding agencies, not CHR and stuff. So they did say that we have been asked for a peak altmetric score and we find it important. Our department, we cannot say for our department,
01:31:20
but we think that this is valuable. I had some excellent, excellent comments. I had a history prof who said that I am not per se interested in altmetrics, but I am publishing a book. I would like to know if there is an altmetric attention score for an individual. When I'm throwing out my book, my publisher wants it. So it's different types of questions and it led to more conversations and explaining
01:31:43
and how that score is calculated and whatnot. And you can go through all the comments. I won't read it, but we are gratified that you're doing this because the research landscape is changing a lot. So thank you for keeping us informed. Some comments about tweets and going viral.
01:32:03
So this has to be a little cautious. We need more information how this contributes to the score and those sorts of questions. That was from Ryerson also, as well as from York. These are the blue sky thing for the altmetric guys. I'm thinking that we have futurity from US, UK,
01:32:22
and Canada, where these universities, and I've marked University of Toronto here, and they take the research articles and then they publish it as news for public to come to know about it. I'm thinking, can altmetric do something like that? We have this small thing of York newsletter,
01:32:41
but can altmetric actually do something for a Canadian context, drawing all this research from Canadian sources and throwing out it on a portal sort of highlighting the university and researchers. But just a thought, LSE and CSH, London School of Economics, Cold Spring Harbors in New York, they have repositories
01:33:01
and they have all these altmetric score. When I'm looking at Ryerson or York or UFT, I don't see that up until now. And this is not that, OK, take the API and put that in and you get that donut over there. I think it requires a little more of hand-holding.
01:33:21
Altmetric, this is just my thought, altmetric, talking to these universities and seeing where exactly can this be actually done because there is, again, so many departments and not everybody would want all these sorts of metrics. So again, there is that too. But I was just looking up this paper on BMC
01:33:40
and the same paper in Ryerson repository, but I don't see that metric. But having that metric, sorting it by highest to lowest, all these can be done and should be done. But again, first you have to talk to the departments and see whether they really want it. This is something I think we do have in Ontario,
01:34:04
the Scholars Portal ebooks platform and the e-journal platform. We do have data like time cited and times downloaded. Can altmetric data be included in that? Again, you will have to talk to the guys at Robarts and see whether the Scholars Portal guys,
01:34:21
the technological infrastructure guys, whether this can be done, should be done. But these are some ways where it can be promoted and there will be a better buy-in is what I'm thinking. And finally, you have PubMed. This is a research coming out from York University, has an altmetric attention score.
01:34:42
We're using it in our teaching and learning. What I've seen is that students, they read an article in PubMed and they don't understand it, but when they read a news article and then they go back to that PubMed article, it makes more sense to them. I've seen that happen quite often with fourth-year undergrad students. From that news article, they come to the researcher who's actually written that article is at U of T,
01:35:02
they get that video too. Then I'm thinking that is there a way to have using PubMed and that altmetric bookmarklet is of course there, but something better where which could be incorporated into PubMed and you could actually sort it by altmetric attention score. I know that there is Lazy Scholar extension which you could install in PubMed
01:35:22
and it will give you the altmetric attention score, but you can't sort it by highest to lowest. So I think that maybe altmetric can step in and actually have some decent bookmarklets. The bookmarklet which you all have at present is a little inconspicuous and you have to really tell faculty members
01:35:41
and students to install it and use it. Again, just a thought. Finally is that I could be talking to people till the cows come home as they say. I could be talking to the university research office, to the media communications officer. What do you think of altmetric and can this be used in your highlighting your researchers
01:36:04
and the university? Nobody talks to me. I think that who is this guy? Shoe disappear sort of thing, I presume. I haven't heard back from the media communication manager till late. I did write to her a few weeks back because I wanted to know if this could be used.
01:36:20
So that is where I think that maybe altmetric could be in the Canadian context, meeting these university people, be it the VPRI or the communication manager and actually seeing what could be done and of course, meeting Scholars Portal and OCAL that is Ontario Council of University Libraries.
01:36:42
I believe we have CIHR speaking about altmetrics. So I will leave it that yes, you all should be talking and working with them, but we'll see what they have to say. Okay, thank you very much. So I'll open it up to questions.
01:37:01
Do you have any questions for any of the panelists here? If you don't mind walking to the mic, sorry. Climb over your neighbor. I had a rather just quick question. What did ARIA stand for again?
01:37:24
What did ARIA stand for again? Oh, okay, Altmetrics for Research Impact Actuation. That was a quick question. Can I follow up Aravind? I thought that was really interesting. The whole session was really interesting, so thank you.
01:37:44
Your specific aria, have you thought about adding gender as a variable or figuring out how to add gender as a variable? Because I see great potential for actually tracking the way in which research gets disseminated
01:38:01
and how gender becomes a factor in that process, which is a hot topic for a lot of us right now. Yes, actually the way we are developing this system is based on the requirements from the university's research support office. So currently they were asking about quantitative data,
01:38:20
the way we visualize quantitative data. So I think the next step we would be looking at is showing more discourse related data to see how the research impact has started and how it has gone over a particular period of time. So I think that's what we would be concentrating next and also trying to add more metrics
01:38:42
related to sentiments in the system. One of the questions I had actually for each of you, if you think of the three different presentations, each of you showed data visualization on top of the metrics and went very different ways of the types of data that you wanted to do,
01:39:01
whether it was putting it in Tableau or even starting with Excel to start with. And I know that the data becomes alive as soon as you show it to someone in a data visualization. It's no longer just flat data. It's something that people react to. So what was a surprising reaction or something you learned from showing the visualizations
01:39:23
to your audience, whoever that might be? You guys want to just like go down the line real quick? Because I think as people are looking at, and the reason I asked the question is I think as we look at the data, we're all trying to make sense of it and find what stories are there. So I think it might be insightful. So you're asking about the surprising insights that...
01:39:43
Yeah, once you could show people the data in a data visualization, right? You built it for a reason to show something. What were the reactions that came back or something surprising you learned? I think they all wanted to see detail levels of data, not just the higher metrics. So for example, if you say tweet count
01:40:00
or if you say an altmetric score of five or four, they want to know where those scores are from. So they want to go to the exact detail level. And the second thing is, they were not really interested about altmetrics because currently it's not used in the university's performance tracking system. So most of them who are not even aware
01:40:20
of their altmetrics scores. So they said that once the university includes altmetrics as part of the performance tracking, they would be interested in looking at it. So for them right now, it doesn't make much sense. Great. In our case, the networks were the most interesting thing for people and since also we were not using a tool,
01:40:43
we built the visualizations custom. So the first response would usually be that can it also do this and can it also do that? But definitely the networks, they grew great in trust. The networks of who's collaborating with whom? Is that what you mean? Yes, feel that. For me, it has been more of a learning experience for myself.
01:41:02
I've become more confident when I talk to faculty and what is going on in the university. Apart from that, I've been showing it to my faculty council and it sort of mirrors to the organized research units we have in our university. So you've got very strong vision research and very strong cardiac rehab with researchers
01:41:21
at University Health Network. So there is that sort of gelling happening there. Great. Question in the back. I have a question regarding affiliation for both Najeev and Aravind.
01:41:41
Aravind, how much do you trust the affiliation field in Scopus when you're reporting results for your institution? I know for our institution and I'm in a Canadian institution, affiliation field is very problematic.
01:42:03
And I know that the work by CWTS, Leiden University, is looking to help Canadian institutions resolve the accuracy of affiliation. So I just would like to know what your thoughts were.
01:42:24
I think we totally trusted at this point. I think the affiliation data provided by Scopus is only for the first author. So we have seen some certain cases where the affiliation data has been wrong, where the countries have been wrongly classified.
01:42:42
But at this point, we are not handling the data quality issue with respect to affiliation. Since we are showing data at an aggregated level, people just might miss it at this point. Running the searches in PubMed, Scopus, Web of Science, because as you said, different researchers use different terms. It could be Center for Vision Research,
01:43:02
but York University is not there. So I'm sort of trying to minimize all those errors by running it in different databases and capturing that. But with ORCID and then maybe there is that, you know, seamless thing which later feeds here. But so it is a miss and catch.
01:43:21
Great, great. And last question, go ahead. Yeah, I noticed, and we do the same thing. The majority of our stakeholders are internally either the PIs or the research operations in terms of business requirements and developing these tools. A lot of data, very complex. To what extent have you extended this
01:43:41
to look at the needs of funders or other types of external stakeholders and how do their needs differ? Because when we deal with them, usually it has to be a lot more succinct. But I wondered what your view was or whether you even do that. Yeah, that's a good question.
01:44:01
I think one of the important stakeholders for this kind of tool is the funding agencies. But we haven't been able to get the exact requirements from these funding agencies because I think they would be more interested to look at where the money they have spent is gone and how much impact it has made.
01:44:23
But at this point, we haven't done anything on that aspect, maybe in the future. Yeah, just in general, the funder data has been very sparse, right?
01:44:42
You get a very limited view if you try to use what's generally out there. At least, we've worked with funders directly and it's really a data collection task always at the beginning where they know who they gave the money to, but they may not know any of the research output
01:45:01
that actually came from that money in a centralized place. So often that's the first place. And all the systems are starting to gather that data even at time of submission of an article to say who funded it. So over time, hopefully all those data points start to link together.