Keynote - Beyond Bibliometrics
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 14 | |
Author | ||
License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/46299 (DOI) | |
Publisher | ||
Release Date | ||
Language |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
|
3
6
7
8
10
12
13
14
00:00
Operations researchData managementFunction (mathematics)Process (computing)Physical systemGoodness of fitStirling numberStudent's t-testMultiplication signMetric systemRight angleUniverse (mathematics)
01:10
Data managementCore dumpStirling numberTime evolutionInternet service providerTransformation (genetics)Observational studyCASE <Informatik>Group actionBoom (sailing)DreizehnLogicFormal grammarGeometryMereologyFinitary relationVirtual machineCASE <Informatik>Observational studyEvoluteGeometryUniverse (mathematics)Endliche ModelltheorieFormal grammarLogicSoftware developerCore dumpBitOffice suiteReal numberChemical equationRight angle2 (number)Wave packetQuicksortProcess (computing)Metric systemMathematicsMassTransformation (genetics)Internet service providerINTEGRALNumberCentralizer and normalizerComputer-assisted translationConnectivity (graph theory)Functional (mathematics)Data managementOpen sourceMusical ensembleRevision controlKey (cryptography)Combinational logicVirtual machineMereologyPhysical lawGame theoryMultiplication signPrice indexSystem administratorCivil engineeringCanonical ensembleXML
08:57
Stirling numberGradientNumberStudent's t-testTime evolutionTuring testInternet service providerService (economics)Scale (map)Universe (mathematics)Scaling (geometry)Endliche ModelltheorieRight angleTotal S.A.QuicksortBit error rateAreaNumberStudent's t-testBitMass2 (number)Degree (graph theory)Internet service providerDivisorMultiplication signProxy serverEvoluteNumbering schemeRevision controlGradientDifferent (Kate Ryan album)Type theoryWeightHypermediaCombinational logicDesign by contractCategory of beingMetric systemRankingPressureAxiom of choiceForm (programming)Parameter (computer programming)Channel capacityAuthorizationPerspective (visual)
18:12
Metric systemSocial softwareData modelStirling numberEndliche ModelltheorieStrategy gameSystem administratorService (economics)Faculty (division)PlanningExecutive information systemVirtual machineBinary filePoint (geometry)MeasurementUniverse (mathematics)Goodness of fitBitFunction (mathematics)CollaborationismOpen setSign (mathematics)Right angleAreaMaxima and minimaQuicksortSpectrum (functional analysis)Process (computing)Decision theoryGreatest elementDifferent (Kate Ryan album)Context awareness1 (number)Mathematical analysisIdentifiabilityMoment (mathematics)NumberTelecommunicationInternet forumTerm (mathematics)Disk read-and-write headType theoryRankingMetric systemPhysical systemSound effectKnowledge representation and reasoningMereologyStrategy gameRoundness (object)Proxy serverLibrary (computing)Data managementMultiplication signField (computer science)WritingStatement (computer science)System administratorBlock (periodic table)Vulnerability (computing)Set (mathematics)RoutingSlide ruleTransformation (genetics)View (database)Real numberDirection (geometry)Interface (computing)FreewareGroup actionNP-hardXMLUML
27:28
Proxy serverVirtual machineComputing platformStirling numberObservational studyUltrasoundMedical imagingFunction (mathematics)ExplosionCross-correlationPhysical systemPerspective (visual)Different (Kate Ryan album)Real-time operating systemTwitterFunction (mathematics)View (database)Strategy gameCross-correlationMedical imagingQuicksortBitObservational studyProcess (computing)Metric systemSingle-precision floating-point formatCASE <Informatik>Cellular automatonSlide ruleXMLComputer animation
30:59
Bounded variationCognitionRankingStrategy gameMetric systemStirling numberAxiom of choiceKognitionswissenschaftInformationDecision theoryMetric systemTerm (mathematics)Natural numberQuicksortDifferent (Kate Ryan album)Video gameComputer animation
32:19
Bounded variationIndian Remote SensingDivisorStrategy gameAnnulus (mathematics)AverageCognitionStirling numberExecution unitMultiplication signRankingKognitionswissenschaftArchaeological field surveyMereologyFaculty (division)Multiplication signRankingStrategy gameBit rateAreaBitPattern languageScaling (geometry)Metric systemQuicksortAverageInstance (computer science)DivisorMassFreewareDifferent (Kate Ryan album)Mathematical analysisUniverse (mathematics)Data conversionDegree (graph theory)Decision theoryTerm (mathematics)HypermediaPoint (geometry)Perturbation theory
36:57
PlanningPay televisionMultiplication signBlogDivisorClosed setOpen setDifferent (Kate Ryan album)Decision theoryComputer animation
37:58
Field (computer science)CognitionField (computer science)Software developerOnline helpPairwise comparisonPhysical systemQuicksortLine (geometry)Different (Kate Ryan album)Metric systemWeight
38:56
AveragePredictionStirling numberGradientData modelTime evolutionMereologyTerm (mathematics)Goodness of fitFilter <Stochastik>BitSystem administratorLibrary (computing)Endliche ModelltheorieMereologyProcess (computing)Universe (mathematics)Perspective (visual)Function (mathematics)Metric system
41:00
TelecommunicationUniverse (mathematics)Video gameEndliche ModelltheorieMeeting/Interview
41:39
Functional (mathematics)Endliche ModelltheorieTelecommunicationQuicksortCentralizer and normalizerUniverse (mathematics)Group actionMetric systemLevel (video gaming)MereologyAreaFeedbackMultiplication signLecture/ConferenceMeeting/Interview
42:47
Level (video gaming)TelecommunicationArchaeological field surveyMultiplication signMetric systemMeeting/InterviewLecture/Conference
43:21
Cartesian coordinate systemSoftware developerQuicksortVector potentialMetric systemLecture/ConferenceMeeting/Interview
44:14
HoaxINTEGRALAreaRow (database)TrailProcess (computing)Goodness of fitLecture/ConferenceMeeting/Interview
Transcript: English(auto-generated)
00:05
Good. Yes, I can hear it now. Fantastic. Thank you. Thanks for that introduction. I it's great to be here I feel like we're friends and colleagues. I've been to quite a few of the AM conferences. I've always really very much enjoyed them and My job at Kings really is to make sure that we've got the best tools and systems
00:23
so I mean, I know most of the vendors and Kings we have pure we have most recently out metrics. So we're a pretty happening kind of place and It was interesting what John said about sterling Transforming young people because 35 years ago, which is probably the last time I was at the University of sterling
00:43
It transformed me. I didn't even study here. I was a serious chemistry student in undergraduate in Edinburgh where things were kind of serious and sterling was where you came for a fun time and To meet really interesting people. So, you know that that's what it's known for is quirky different and
01:02
Transformational that's for sure. Anyway to serious things. I need to find my squidger Let's see if the squidger works. All right So here's what I do. I've got a kind of interesting job at Kings, but I really am where the rubber hits the road and as the function lead I provide all of the kind of central support for
01:26
research at Kings College So that includes the typical kind of research have been the awards management stuff Commercialization clinical trials we do enormous amount of clinical trials at Kings College It's the post-grad scholarships the compliance all of the animal human ethics GDPR research integrity research governance
01:47
the core facility stuff And all the research development stuff and of course research developments the important competitive component that we're all very interested in So there's about 600 people covering in my portfolio covering every aspect of central support for research
02:05
So I know the research bit, but I'm not on the academic side. I really am trying to Increase our performance at Kings as one of a you know top Russell University in the UK So that's me
02:21
Cat asked me to talk about you know, how do we use All metrics in At Kings and I thought before I do that, I think there's a bit of history that we have to look at There's a transformation going on in the university world that I think is really important that we understand now maybe many of you understand it, but
02:41
It's a topic of much debate around the world. And of course governments want to put money into a transforming university That does the right thing for society So this is kind of what I wanted to talk a bit about today the evolution of university through the ages Won't talk too much about that the changing role of the university. I think that's important
03:08
The evolution and diversity of the research provider which isn't always a university the transformation of metrics needed to describe these different Pieces that we're trying to demonstrate our ROI to the government on and
03:24
Then the bit that cast for how and who uses our metrics at universities And I'll just do a few case studies and I'll just say up front. Thanks to Jurgen at altmetrics who helped us and actually really did such a good job in persuading the administration
03:41
At Kings that was a good tool and how he did that was he made a very sort of live Here's what Kings looks like through the altmetrics lenses It was very useful case studies that brought it really alive for us at Kings So here's the history I'm not sure if you know, I didn't really know I knew about Bologna but between Bologna and today
04:03
I didn't know the details. So this is what a history of University Thousand year actually even more than a thousand year looks like so the first sort of Indications of anything like a university were in the med medieval ages and these were studia where were set up to educate clerks and monks
04:20
ancient history I suppose the first inkling of what today's University looked like was in Salerno in the 9th century and it was a medical school that was created to teach Physicians or in those days. I think they were called Bar surgeon barbers or barber-surgeons. It wasn't physicians
04:41
I Think the first time there was a true University was in Bologna I think we all agree that they were the first in the game in the 11th century And they were very focused on Canon law and civil law and it was cool to be in Bologna for Iyama at that very University. I don't know how you felt. I felt I thought that was a real
05:02
Exciting privilege to be there Ben was there as well. And so I recognize some of the rest of you were there too Okay, moving forward a little bit to the 12th century then a couple of key universities were set up Paris and Oxford Very focused on theology and then quite, you know theology and I was gonna say dogma that might be a bit
05:24
Insinutive so I won't say that And then between the 12th and 18th century, I think there was There was some development and then it ended up with what's often called the Scottish enlightened version of University And that's the kind of liberal arts
05:42
Personified Rome Heidelberg's and Andrews maybe Glasgow as well offering a core curriculum liberal arts grammar logic rhetoric Geometry arithmetic astronomy and music's quite a quite a nice combination And I think most recently we all recognize the sort of German model of
06:04
University which 19th century typified University of Berlin and that's where experimentation and the scientific method sort of replaced Conjecture and and and sort of Theological dogma, right? So that's kind of the beginning of the new age of history
06:23
so that's just kind of a evolution through the ages and Where we've got to today roughly All right, so where does that leave us looking at sort of more modern history I think the old model was definitely a university as a womb as a crucible of societal knowledge and new discoveries, right?
06:45
that's that's pretty much where some of us still are but Where history got us to? But I think that role has really changed We're now seen as the sort of fountain of high-quality Personnel and There's a very strong training agenda that the universities have been given and that's a risky one because I think
07:05
And we'll talk about that a little bit more and I'm interested to hear from you But I mean when I was a young lad there was a very small number of people got to go to university It was a small percentage And that has changed completely, you know I think in those days it was people that were just interested in learning and it was an opportunity to become a very rounded
07:25
Educated member of society that has changed. I think we present ourselves now is much more of a mass training kind of opportunity Then there's a role in as part of the economic development machine and that's huge for us. So that's
07:41
that's sort of messing around with the balance between pure and applied research and dabbling with industry and You know the metrics around Spin-offs and and licensing all the rest of it and then I think the most recent one probably last ten years or so I think the commercialization one and the
08:01
Economic development one started at the end of the Second World War But really hot heated up about 20 years ago where there was a lot of money put into our industry liaison offices and there was a real ramp-up of commercialization and spin-offs The most recent one probably last ten years is kind of broader societal impact that's going beyond kind of just economic return
08:24
That's how do we influence public policy? And how do we improve public health and Go beyond just making money. So that's that's the most recent one But the question really is can we satisfy all those roles? Those are big roles in society
08:40
And and do we all need to do that? I think another question is we have elite universities We've got medium-sized universities. We've got small regionally relevant universities and everything in between So that's that's the role and I think most of us are some combination of all all of these the crucible of knowledge the fountain
09:04
Of HQP the engine of the economy and the societal impact Agenda, and I think in some ways what you could do is sort of classify a university by weighting these four different Aspects, but I will say one thing about impact. That's really important that John kind of almost hinted at but
09:27
So in the ref and I think many of us think about impact as well, that's citation impact, right? That's that's that's the definition of research excellence and impact but interestingly the Australians did a pilot where they tried to create a similar to ref type of
09:44
Funding scheme But instead of having peer review as we do in in in the UK to judge how much impact our research is having The Australians decided to go with non you peer review. And so they they had industry they had government they had just Taxpayers and the one thing I'll tell you that is really interesting in that pilot that they did
10:05
was that our version of impact is completely different from general societies and so We can't even agree on what positive impact is when it comes to research I think that's really important to understand as we think about the metrics that define success, right?
10:24
Okay, so let me tell you a bit more history to give you a few numbers and give you an idea of the evolution This is just the number of universities in the world and you can see there was very few in in the 1400s slowly ramping up 16 25 to 73 the end of the Second World War there was 500 today this
10:45
10,000 so, you know we have grown like Topsy all over the world Some massive increase in number of universities and of course We have a global ranking systems and we're all desperate to get up those global ranking systems
11:01
Shanghai and Times Higher Ed and I think if you look at those recently and we worry about them a lot Some countries really are doing a great job of whipping up those ranking systems and we'll talk about that a little bit further Here's another one that's really important is how much money Your government spends on R&D in universities as opposed to in other places
11:23
So good is the the total expenditure in R&D, but that's made up of herd which is higher ed spending Bird which is business expenditure on R&D and then at gird at not gird governed Which is government spending on R&D
11:40
But it's important for us because the more money that obviously we spend on herd, you know that that that makes us Busier, but what you can see here is that countries like China Japan and US do a very small proportion of their research in Universities of the total spend right that's very significant on the other hand the UK and Scandinavia
12:01
Do a third of their research in university total third of their total research in Universities now the sort of pros and cons I suppose one would are the Scandinavians would argue and the Brits would argue Well, you know, we're going to do all the disruptive research and make the big discoveries in universities. Not an industry I don't think that's necessarily true, but that's that's one argument
12:24
On the other hand, I think in the US they would say well because we're working so closely with industry The discoveries are closer to the market and the more useful for the economy and for society, right? So it's kind of a dual argument there Sometimes though we don't have enough industry and we're kind of forced to do lots more heard almost as a proxy for missing
12:47
industry R&D so for Canada's like that. We don't have a lot of
13:04
Universities transform. Well, we're educating more and more students. It used to be a sort of a 5% of the Population and now it's 50 or somewhere between 25 and 50. I know Tony Blair said 50, right? I think 50 is way too much but most Western countries are sort of between 25 and
13:24
50% of all young people are expected to get into university and have some sort of tertiary education Fulfilling all the roles we've talked to our educator creator of knowledge worker trainer engineer economy provider of broader societal impact But the most important thing is we're soaking up more and more of the GDP
13:44
And we have to justify that we're costing more. We're promising more and we have to Show up we with what the ROI is and all these different roles. That's really where the pressure is alright, so
14:01
Let's talk about the research providers because I think this is a really important factor as well now if you try and classify them into a taxonomy There's four or five different versions and I say providers because it's not always universities so we got the comprehensive university and
14:20
Most universities I suppose fall into this category where they teach a massive number of undergrads and who pay the bills Really, especially in the UK So we have a lower number of grad students lots of researchers, but covering a huge research landscape needed to provide all of the courses to teach all this massive number of
14:42
undergrads in many different disciplines right now What that means is that? There'll be a low number of areas of prominence. So if you use the Seibel Lens and you look at those universities what you'll find is a relatively low number of areas of prominence because there's such a spread of
15:02
Research expertise right a bit of a downside, but a high degree of curiosity driven research And then you've got the elite research intensive universities. They have lower undergrad numbers high number of grad students more but much higher Areas of research prominence and when you do the Seibel look at this you'll see that the these elite universities have
15:24
Sort of 80% of the research is in areas where they are really world-class Whereas the comprehensive universities, it's somewhere usually between about 10 and 20 percent Right. So that's a major difference is is critical mass and nowadays research is a team sport and this critical mass turns out to be
15:43
Really really important if you're going to have world impact. I don't think there's The model of the one PI with the one grant doing world-class research still exists But but I think the team sport piece is is taken over then the third one I want to mention is the elite research provider and
16:02
these are Not universities, but they're sort of dominated by universities things like the Crick Institute The newly formed Turing Institute the catapults in the UK I'm not sure what these similar things look like in in other countries But they're definitely dominated by the university. They're created by
16:21
governments and they're supposed to Supercharge our research activities often with partnership, you know, the health authorities and Industry, but they're special crucibles Unencumbered by undergraduate education and then the fourth one the fourth one I want to mention is a very special one and
16:40
That the Germans really invented and are very very good at this is a Fraunhofer Institute's where? They're not really dominated by universities at all. In fact, I would say they're more dominated by Industry and they're largely funded by industry and they're largely contract research. So these are our Research institutes that really focus on what are the problems that industry and governments are trying to solve?
17:05
so the most pure form of solving the exact societal problems Just a bit more detail on Fraunhofer, so this is about Huge scale to 72 theme themed institutes across the world 26,000 people
17:21
2.6 billion euro budget and interestingly of that number 2.2 is contract research So 2.2 directed research that's huge To give you an example in the UK I mean most universities are doing about sort of five to ten percent industry funding and Some of that will be contract research. So completely different scale of contract research
17:44
Alright, so that's what the models and why is industry so important? Well other than Promoting universities as sort of engines of the economy even from a very selfish PI perspective when you have co-authored papers with industry your impact goes way out citation impact goes way up and
18:03
impact and Social media attention everything goes way up. So you get huge bang for your buck But again at King's that's about five percent of our publications are co-published with industry It's a small number and you can see, you know The sole authorship is way at the bottom in terms of impact this so this is outputs in the top 1% citation
18:24
Percentile and again the corporate ones at the top sole authorship on the bottom as you might be able to predict Alright, so let's talk about metrics now. So now we get into the metrics piece I think most of us sort of have an idea that citation impact has done some good stuff and has been a reasonable proxy for
18:44
Research excellence but not for all of the research spectrum So it doesn't work very well for a lot of social scientists or humanists And in fact, it doesn't even see them because they don't publish in traditional scholarly journals
19:01
The fountain of HQP, you know, this should be really easy and but we don't do a very good job of it In fact, you know when we think about our graduates, we don't really care that much about them until they become You know older and richer and then we think of them as alumni and we phone them up and say, you know How about giving us some cash and it's big business
19:20
I mean at UBC we raised about 200 million dollars a year from our friendly alumni and rich donors and stuff So it's big stuff The economic engine bit I think we've got kind of we know what the metrics are but the societal impact is the one that really really is difficult and We're really going into this area hard. So at Kings, I mean we spent 30 million quid
19:44
Recently to buy this science gallery. That's right beside guys. It's like a fabulous Monument to you know, it's kind of a gallery And what we do is every three months we have a new sort of research theme and it's free There's a big sign saying come and visit for free. There's a cool coffee shop there
20:04
as I say, it's thousands of people walk by it's right beside Guy's Hospital and right beside the shard a Different research theme every three months. The last one was dark matter But it's not just about science. It's about art. So it's about the science and art of around dark matter at Kings
20:23
The one before that was called hooked and it was about addictions the science of you know, Kings is big into addiction Research and it was about you know art of people who are addicted to so it's great and and we even have a VP called the VP London
20:41
Who's in charge of public engagement? So we spend huge amounts of money on this Baroness Deborah Bull is our VP London who? You know runs a science gallery and all of our kind of public engagement outward facing stuff So it really is very big for us. But how do we measure this? We absolutely fail to measure effectively To this point and I think this is where the big opportunity for altmetrics really comes into play
21:07
So the bibliometrics works fine for some of the crucible and knowledge model, but absolutely not for the societal impact and So, you know, this is this is where we look at the transformation of how we use metrics
21:21
Okay, so here's what can't ask, you know who's using altmetrics in Universities well as I say I represent one piece of that and that's kind of where the rubber hits the road at universities in the Interface between researchers and the real world researchers and funders researchers and industry and so what the different bits of
21:42
What we do are Things like, you know, our vice-president research Reza Resave Worries about things like what's our research performance? What's our strategy Who do we partner with those are the big sort of executive research questions and they are informed by
22:03
You know our out metric and our side valve view of the world People like us the research administrators. We worry about research excellent metrics ref is absolutely huge and I'm going to show a few slides further on that really show how Valuable our metrics is in doing a better job of ref
22:23
so the way ref works for non Brits is that there's a huge amount of block funding available and we have to do a Submission and that submission means that we have to present the best papers for each academic at the institution And it's all about impact. They get one the two or three or a four star
22:43
And then we have to write Impact statements for a limited number of our researchers and again There's a sort of a peer review of all of this stuff and at the end We get a big check and we get a bigger check if we're more competitive than our peer Universities the last time I think we did really really well. We're not quite sure why but this year
23:05
This round we're going to do really well, and I think it will be Largely because we really worked hard at identifying our impact and Evidencing our impact and we've gone beyond the usual kind of idea of field normalize citation impact and stuff like that
23:23
Librarians, I mean, you know better than I do how they use Metrics, but I will say at King's our library system. It's very important because they are Almost holders and protectors of the systems and tools and trainers of people So when it comes to things like our effectiveness at using CyVal and altmetrics
23:42
The librarians play a huge role in making sure that that sticks and works well at King's And then there's a traditional kind of you know your department heads that worry about things like assessing their PIs and who gets promotion and who gets tenure and And that type of stuff, you know the basic kind of performance management
24:04
And then the big the big one I would say that really it is a huge opportunity for us is the PR and communications Which is really big we want to be noticed. We want to build our reputation and we do that by Find our way into the news and into important public fora
24:24
So that's a bit of a you know, that's a bit of a sort of a mishmash of where the opportunity lies I don't think King's has taken full advantage of that. We just started subscribing fairly recently We're sort of still getting our head around what the value is if you ask me what's the biggest win for us? It's absolutely reff at the moment
24:42
That's where the value is. That's how I sold it to the decision makers that that provide the big money actually, it wasn't I had to convince myself it was a good idea, but So this is what we do identify research impact
25:00
Evidencing research impact for ref I think importantly providing qualitative context to the bibliometric analysis of our outputs at King's is important as well research strategy We worry about partners, you know We universities join clubs we have big global clubs of universities in the UK the clubs called the Russell group and it's just the biggest and best but
25:26
internationally their clubs like the u21 UBC was a member of the u21 There's one called the guild that we're a member of at King's It costs a lot of money to do this and you want to make those decisions about which club to align with
25:42
sensibly so that there's maximum opportunity for collaboration as you invest this money in partnering with your club members Assuring and justifying open access talk about that a little bit more and Evaluating obviously the non-stem research outputs is very important for us as well
26:02
So we know what the pros and cons are, you know the bibliometrics It's a great direct measure or a good proxy for research. Excellence is very important for global rankings But it only sees certain parts of the research outputs the research spectrum I think the altmetrics is an opportunity for more nuanced approach to measure an impact. It's a more direct
26:27
measure of impact it sees Research activities in social sciences and humanities are really important I mean UBC our social scientists were higher up the world ranking than our medical researchers And so this is this was really this is really important for us
26:43
But you know on the con side It does measure negative attention as well as positive and I think the biggest chink in the armor is that You know unlike citation impact sometimes the data sets are not comprehensive and so it misses stuff and that's because
27:02
things like mentions and policy documents it's about the Scrapability of those and accessibility of those documents and so but you know Researchers if their paper isn't picked up by a particular system the researcher Immediately thinks well, this is a crap system. It's missed me out. So that is a dangerous
27:25
vulnerability of metrics that I just mentioned as obvious So this is this is what it's all about. We all know these different aspects of metrics I should mention that plum X does slightly different stuff and you know, I think
27:44
These systems are very complementary and I think It's not a question of either or I think it's a question of Just making sure that you've got access to all of them or most of them or as many as you can afford And make the best possible use out of them
28:02
So here's a few case studies And I wanted to convince myself and this is the work of Jurgen that our top researchers from a citation impact perspective We're also top researchers from a metric pure out metric look at the world. And so Seb is one of our top Researchers, he's a medical imaging guy. I think this is a three-year snapshot 400 outputs
28:29
240 of them are picked up in the altmetric view of the world 1800 Twitter 48 pan 160 news mentioned so, you know shows up loud and clear in both systems and that's that's that's good to know
28:44
Here's some of his top Top spotted papers real-time imaging of single neuronal cell apoptosis in patients with glaucoma is important stuff, but he does show up really well in both lenses And I think that was important for us
29:03
This is Reza, this is our vice-president research similarly Reza is a pediatric cardiologist who Treats very young kids and he's trying to improve the scanning Capability so that he can do a better job of diagnosing and and treating
29:21
congenital heart diseases in very young kids Interestingly asked him where is his his drive to do this came from and he said well I have a daughter poppy who has a congenital heart disease and he thought this was worth going after so that's his personal story Which I thought was very inspiring And again, he shows up in the altmetrics view of the world as well
29:43
Thanks, Jurgen. All right, and Let me just sort of talk a bit about When you sort of compare and contrast, right? This is a this is work that we did at Kings looking at 144,000 Kings journal articles
30:02
and so on the x-axis you've got the altmetric view of the world and on the y-axis you've got the Citation impact view of the world now you look at that and go it's a bit messy it is But you know, there's a correlation. I Think the most important thing though more important than correlation
30:21
is that if you want to pick up all of your impact and Attention right the the the citation impact is going to give you this stuff Both is this stuff so you've got some correlation there but what about all this stuff, you know, this is stuff that you're going to miss if you don't look through the altmetric lens and
30:44
We can't afford to miss this. We can't afford to miss it for ref. We can't afford to miss it for our You know our outward facing facing public engagement strategy. So this is really important I think this is an important slide and not to be forgotten
31:01
Okay, we life and death is is is all about publishing for researchers and where they publish and so Everyone wants to publish in the highest You know impact journals and of course things like the Shanghai global ranking One of their key metrics is how many papers do you have in nature and science?
31:22
So, you know, that's how serious this stuff is but I just wanted to give you a couple of things that we look at so here's a different publishers and different journals and This is the five top publishers in psychology and cognitive science and you can see that They vary very widely in terms of news mentions per article now. Why is that important?
31:46
Isn't it all about citation impact? Well, no and There's lots of reasons that this is important when it comes to news mentions, you know things like donors publish donors and for performance
32:01
Reasons as well. It's really important to know what sort of attention you're going to get in different publications You can then put that information beside your citation impact and make clever decisions on where you need to Publish if you so if you're lucky enough to have that kind of choice
32:20
same kind of thing This is the five best journals in psychology You can see the journal impact factors, which of course we're never supposed to even look at but we all do And what you can see there though, interestingly is the altmetric attention score per article is Almost 180 degrees apart right the top
32:41
journal in terms of impact factor is the lowest on the metric attention score so Interesting what you can drive out of this kind of analysis, you know, it's not all obvious That the top gif is the top place to get attention
33:00
Outside of the scholarly community and as I started my conversation today You know scholarly impact isn't the same as real impact or when I say real impact what society thinks is real impact Again here's more of the same. This is the top journals in psychology and cognitive sciences average news mentions
33:21
And so again, this gives you the massive scale of difference between the different journals and what how likely they are to get picked up and and Picked up by the the media This is a super important one as well. So when I think about public engagement, I think of it. This is
33:42
This is the Scopus topic at Kings Receiving the most news over the last three months. So this is kind of what's hot at Kings Why is this important? well this is what people are talking about and want to know about and and need to know about and so that
34:00
Informs a lot of different things at Kings like, you know, what's our externally facing PR strategy for this month and what sort of Externally Engagement activities what kind of lectures are we going to be doing? You know if we want to be relevant and we want to be Up front. This is this informs that exact strategy Baroness bull wants to know what this looks like and what to react to
34:26
It's no and interestingly some of these like for instance tobacco, you know, that doesn't show up in in the citation impact because It's not stuff that we publish on but
34:41
The public are sure very interested in it. So there are areas that really You know diverge when we look at our traditional scholarly assessment of quality This is policy mentions. So again, you know, why is this important? Well, we've got a policy Institute that needs to know
35:01
What areas of research that we're doing at Kings are the most important I'll mention another thing and I shouldn't really say this But we do worry about it Is that there's quite a bit of research recently on funding patterns and it turns out that your success rates are way higher If you are asking for money in sort of trendy areas of research now
35:21
It's probably no surprise to anyone but you know, it's important to know what those trendy areas are So you might ask well, how important is it really to be in the news? Why don't we just ignore it? I don't think we can for many different reasons
35:41
If you think about Times Higher Ed ranking as an example a huge part of that is the subjective Global survey they send out 300,000 surveys to researchers all over the world Who publish in their journals and and ask questions about you know? What's your ranking the best research and the best teaching universities in the world?
36:01
Now I've talked to some of those researchers who fill in that survey and I've said, you know, what do you think about? How do you make your decision on the best universities in the world? Well, there's three things Usually they've had a personal experience You know, I was a postdoc at Oxford wonderful University and I'm of course, I'm biased. I think that's a great place
36:20
There's history and prestige universities like King's who've been around for a while And we have I think we've our medical faculty Began, you know 900 years a long long time ago So we get some free free prestige free
36:41
Points for being old and prestigious and then the third one that's really important always is said is you are in the news You know, we hear about you. You're obviously doing relevant research Society cares about it and that's important. So it's kind of those three things. So I think absolutely it's important I'm getting close to the end of my time here, but I'm almost finished
37:02
Open access. This is really important for us. It's coming down as regulated Plan S has been delayed by Europe, but it is coming The reason it's important is that the publishers charge us twice for a lot of stuff, you know They're charges for a subscription and then charges for an APC to bring a piece of research
37:22
outside the paywall and so we have to make different difficult decisions about where we publish whether we can afford to publish in certain Journals, but this is just so we have to convince ourselves and convince our funders that they should fund APC's so that we can do this open access stuff and you can see the difference between
37:42
Closed access and open access is huge. It's a factor of three for altmetric attention It's a factor of five for news mentions Factor of four for blog mentions, you know, so open access we can demonstrate the value of that If we're not doing open access, we're not being noticed Similarly here, this is the field normalized citation impact
38:05
This is the funder. So this is the Wellcome Trust Funding and what you can see here is that the research funded by Wellcome Trust Gets a lot more attention gets a lot higher citation impact attracts a lot better science compared to
38:20
Non Wellcome Trust funded again. I can use this and say researchers. You should be applying for welcome I know it's hard to get but you know It's going to be worth it. And by the way, I'll help you get it with various research development activities this is a side-by-side comparison of field weighted citation impact and altmetric attention scores on
38:41
The same issue Wellcome Trust funding and you can see two things big difference between Wellcome Trust and non Wellcome Trust but you can see nice sort of lineup of both systems altmetrics and citation impact I think it's the last one When we look at REF, like I say an important thing for us is to be able to predict
39:04
How good our papers are in terms of one star two star three star four star. We want everything we submit to be four star We use citation impact We use a bit of internal peer review and we also look at the scores here Now you can say wow, that's you know, there's a lot of overlap
39:22
Yes, there is but as one of a few different filters to put our outputs that we're considering for REF submission You know this makes sense to look at All right. So what if we learn and I'm a minute early I think The university model is evolving absolutely evolving and it really is changing
39:43
The way we use KPIs and the KPIs that we need to use I think I've shown a little bit that altmetrics is highly complementary to a traditional It does the same kind of job and it does a different very valuable job for us in universities I think it's the impact agenda which is very important in the UK, but certainly in North America as well
40:06
absolutely demands These new ways of measuring impact and the ROI just can't be done with with the traditional bibliometrics And and and the last thing, you know altmetrics is used by and can be used by
40:21
many parts of the institution for a very effective Outcome I think those are the main things I want to say thanks for listening. I hope I hope you learned a little bit and Certainly, I know this is a diverse crowd of people. I don't know how many other research administration people I know it's libraries and publishers. But yeah, okay good. Well, it's good to see some colleagues in the same business
40:45
I hope the rest of you have learned a little bit about you know Research administration and how university works from our perspective. Thanks very much for the opportunity Before we let you escape any questions that anyone would like to ask
41:11
Martin you refer to I made a note of it PR and communications as becoming a more important aspect of university life as well
41:21
How does that work out in King's College? regarding bibliometrics feeding back into PR and communications Normally that those are two different Departments are they teaming up more than before or how's it going? So I'll tell you interestingly Kings has made a big bet on a completely different model of how we run the university
41:44
It's called functional alignment. What that means is that There's a function lead that manages every aspect of a particular part of the institution I'm research now what that really means though And so communications is a function what it means though is that I mean I have communication people in my area
42:04
But all of them have a strong direct relationship to the function lead in communications So you could think of them as moles or spies or whatever in research, but we have a level of communication and cross
42:20
cross disciplinary kind of communication That allows that kind of thing to happen so the communication in my group Feedback into the central group with things like these metrics all of this stuff I'm talking about so we have sort of an extra layer of
42:40
Channeling built in through this functional alignment at King's that's probably the best way to answer it I think it's a big advantage. It's not easy to do and it's costing us lots of money and time There's a dip in You know the satisfaction surveys of our researchers because it's it's it takes time to get right but that's that's kind of what we're aspiring to and that's that's where we're seeing the
43:05
Level of communication happening. Okay, I think that's very good to hear You were also talking about this negative attention that may be something that may work out as negative Towards all metrics as well. How do you deal with that in King's College?
43:21
you know, I was I actually remember a couple of years ago asking you and about that and I have to admit that although it's sort of the witch in the cupboard I don't think it's that big a deal But I seem to remember you and saying that what the industry was looking at was there are clever a potential AI
43:41
Applications where you can screen out and filter sort of quite easily. It's not that difficult to do So I think that's one of the traditional sort of here's why we don't like out metrics, but I actually think it's almost a non excuse nowadays and You know the way AI is going so I've got AI Applications that can sort of do research development do very clever things
44:04
So I actually I think that's almost not worth mentioning anymore to be honest with you. Hi Joe from cross ref
44:22
Do you care about fake news articles? There's a lot of there's a lot of news articles out there Is that not exactly the same question? I think it is and I don't know If I know the right answer to that and you know how sophisticated is Kings at spot in that? I don't think we are very sophisticated
44:42
But we do worry about a lot and you know, if you expand the fake news to fake science You know one of the things we worry more more most about is research integrity and you know You've probably heard about some of the breaches that have happened in the recent past that really have tarnished University reputations and brought down presidents and vice-president researchers Karolinska
45:06
Paolo macarena is probably a fairly recent one. So a fake news and fake science Absolutely deadly for us and I you know, we worry about that a lot again We've got good process in place to handle some of this stuff, but I don't think we've got a track record of successfully
45:24
Protecting our reputation in these areas