Jonathan Grant, RAND Europe at the DataCite summer meeting 2012

Video in TIB AV-Portal: Jonathan Grant, RAND Europe at the DataCite summer meeting 2012

Formal Metadata

Jonathan Grant, RAND Europe at the DataCite summer meeting 2012
The science of science: Why measuring research is necessary to improve the way it is funded
Title of Series
Part Number
Number of Parts
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
10.5446/8391 (DOI)
Release Date

Content Metadata

Subject Area
Performance appraisal Modal logic Computer animation Lecture/Conference Different (Kate Ryan album) Gender View (database) Video game Perspective (visual)
Complex (psychology) Message passing Word Meeting/Interview Civil engineering Multiplication sign Sampling (statistics) Matrix (mathematics) Game theory Physical system
Group action Characteristic polynomial Mathematical analysis Mereology Event horizon Plateau's problem Number Independence (probability theory) Meeting/Interview Reduction of order Office suite Information security Multiplication Focus (optics) Decision theory Online help Length Mathematical analysis Internet service provider Independence (probability theory) Mereology Performance appraisal Word Process (computing) Computer animation Telecommunication Freeware Table (information) Near-ring
Computer font Observational study Observational study Mathematical analysis Field (computer science) Mathematical analysis Mereology Event horizon Goodness of fit Performance appraisal Process (computing) Resource allocation Computer animation Personal digital assistant Personal digital assistant Order (biology) Universe (mathematics) Video game Traffic reporting Resource allocation Near-ring
Group action Context awareness Dynamical system Multiplication sign Execution unit Set (mathematics) Parameter (computer programming) Disk read-and-write head Meeting/Interview Personal digital assistant Logic Software framework Endliche Modelltheorie Resource allocation Exception handling Physical system Area Algorithm Observational study File format Measurement Type theory Process (computing) Website Endliche Modelltheorie Ranking Quicksort Freeware Point (geometry) Divisor Number Lecture/Conference Well-formed formula Touch typing Representation (politics) Biostatistics Dependent and independent variables Scaling (geometry) Matching (graph theory) Information Archaeological field survey Projective plane Mathematical analysis Paradox Basis <Mathematik> Limit (category theory) Performance appraisal Personal digital assistant Logic Universe (mathematics) Video game Table (information)
Observational study Multiplication sign Mathematical analysis Focus (optics) Perspective (visual) Theory Performance appraisal Meeting/Interview Different (Kate Ryan album) Personal digital assistant Logic Matrix (mathematics) Message passing Multiplication Observational study Constraint (mathematics) Texture mapping Key (cryptography) Archaeological field survey Mathematical analysis Message passing Process (computing) Resource allocation Computer animation Personal digital assistant Logic Endliche Modelltheorie Key (cryptography)
Point (geometry) Multiplication Multiplication sign Parameter (computer programming) Computer programming Theory Dimensional analysis Wave packet Product (business) Performance appraisal Arithmetic mean Performance appraisal Different (Kate Ryan album) Key (cryptography) Object (grammar) Series (mathematics) Message passing Multiplication
Computer virus Point (geometry) Sine Free group Observational study State of matter Multiplication sign Control flow Time series Insertion loss Number Goodness of fit Estimator Causality Meeting/Interview Lecture/Conference Term (mathematics) Software testing Endliche Modelltheorie Series (mathematics) Office suite Traffic reporting Multiplication sign Raw image format Observational study Electric generator Information File format Software developer Physical law Stress (mechanics) Sound effect Bit Line (geometry) Message passing Process (computing) Personal digital assistant Universe (mathematics) File archiver output Website Digital Equipment Corporation Object (grammar) Game theory Exception handling Library (computing)
Email Sensitivity analysis Observational study Confidence interval Weight Multiplication sign Division (mathematics) Online help Total S.A. Event horizon Hand fan Field (computer science) Number Frequency Estimator Type theory Bridging (networking) Different (Kate Ryan album) Core dump Determinant God Exception handling Addition Inheritance (object-oriented programming) Electronic program guide Mathematical analysis Generic programming Reflexive space Bit Price index Mortality rate Number Frequency Computer animation Personal digital assistant Video game Cycle (graph theory) Game theory Pressure
Point (geometry) Multiplication sign Focus (optics) Observational study Observational study Connectivity (graph theory) View (database) Interior (topology) Translation (relic) Water vapor Bit rate Mortality rate Mereology Sound effect Number Estimator Computer animation Meeting/Interview Different (Kate Ryan album)
Sensitivity analysis Statistics Group action Observational study Weight Multiplication sign Video game Design by contract Perturbation theory Insertion loss Mereology Mortality rate Number Causality Natural number Endliche Modelltheorie Pairwise comparison Traffic reporting Area Multiplication sign Service (economics) Execution unit Observational study Inheritance (object-oriented programming) Coalition Gradient Bit Basis <Mathematik> Mereology Lattice (order) Process (computing) Computer animation Universe (mathematics) Video game Spacetime
Point (geometry) Building Digital electronics Observational study Multiplication sign Source code Online help Perspective (visual) Number Frequency Medical imaging Mathematics Performance appraisal Meeting/Interview Different (Kate Ryan album) Hypermedia Single-precision floating-point format Software framework Traffic reporting Arc (geometry) Multiplication Formal grammar Data type World Wide Web Consortium Area Metropolitan area network Channel capacity Projective plane Sound effect Coma Berenices Mereology Complete metric space Flow separation Type theory Category of being Pointer (computer programming) Computer animation Personal digital assistant Strategy game Different (Kate Ryan album) Website Software testing Pressure Form (programming) Local ring Relief Surjective function
Point (geometry) Trail Multiplication Statistics Digital electronics State of matter Building Digitizing Software developer Channel capacity Collaborationism Interactive television Set (mathematics) Multiple Regression Database Drop (liquid) Cartesian coordinate system Type theory Computer animation Right angle Formal grammar
Observational study Execution unit Archaeological field survey Insertion loss Mathematical analysis Streaming media Graph coloring Number Revision control Meeting/Interview Different (Kate Ryan album) Moving average Nichtlineares Gleichungssystem Series (mathematics) Addressing mode Resource allocation Maß <Mathematik> Mathematical analysis Sound effect Basis <Mathematik> Bit Total S.A. Type theory Moment of inertia Personal digital assistant Infinite conjugacy class property Different (Kate Ryan album) Arithmetic progression Physical system
Observational study Divisor Multiplication sign Mathematical analysis Focus (optics) Dimensional analysis Number Data management Frequency Bit rate Meeting/Interview Different (Kate Ryan album) Personal digital assistant Logic Series (mathematics) Data structure Scale (map) Observational study Information Building Software developer Archaeological field survey Mathematical analysis Basis <Mathematik> Mereology Peer-to-peer Resource allocation Computer animation Personal digital assistant Software framework Endliche Modelltheorie Faktorenanalyse Right angle
State observer Multiplication sign Archaeological field survey Perspective (visual) Dimensional analysis Data management Mathematics Bit rate Meeting/Interview Personal digital assistant Software framework Resource allocation Position operator Area Collaborationism Observational study Electric generator Building Sound effect Database transaction Special unitary group Mereology Measurement Data mining Software framework Normal (geometry) Website Web page Point (geometry) Identifiability Observational study Divisor Online help Mathematical analysis Number Software testing Traffic reporting Scale (map) Multiplication Matching (graph theory) Information Expert system Paradox Peer-to-peer Performance appraisal Resource allocation Personal digital assistant Universe (mathematics) Statement (computer science) Faktorenanalyse Greatest common divisor Game theory Object (grammar) Family
By thinking that Introduction and would give further Invite me to participate in this conference I think the candy gender I'm going to give you a very Different view like perspective on research research evaluation dance on the discussions you have later in the day said I hope that's useful life it stimulates thinking And they stimulate some ideas for you as a community it ain't going forward and Harry you are going to demonstrate the value of what we're doing because that's
Fundamentally what Politicians and civil servants right across Europe North America and the close Increasingly asking Research communities and in times of austerity and you're crises and financial meltdowns those questions become even shot in the policy of dialog but I'm going to do is to give you a brief A review of why I think it's important to understand research impact on my will then give you 2 examples contrasting samples both coming from the biomedical health care and giving this teachers illustrates the logical complexities but also to illustrate that you can't do it he tried as questioned potent message to get across some 1 . 2 Definition will use the word impact them quite a lot
Autoimmune by impact say I don't mean by impact I don't mean much paper having paper cited academic impact that's when you have The impact when your academic colleagues that impacts entirely Valley and a lot of tools and measuring the impact of baby matrix which were reasonably well and biomedical sciences in the hardest is less violent human Donald Akin that academic impact on the game impact BU wants the academic system so as people who will largely funded by taxpayers euros dollars for his sterling What are you doing to make society better place that impact can occur in many many different ways
Stanley just give you a quick background rent Europe so you have an understanding of where I come from we're Independent Multiple that public policy research institute that is a helpful but actually it describes us but we were not communications people went a step below that wanted to byways Aziz of think it sums it up quite nicely and we have mission which help improve policy-making free research analysis with very very applied when it comes to doing research and I would like to say the single most important word that mission is hell we yeah we know that research analysis is not a technocratic solution policy-making that we can reduce season that policy process in that it with dialog but he evidence on table getting engaged them with delivering on our mission part of the global Rand Corporation which is a not-for-profit based on U.S. increasingly set up offices in the Middle East and most likely in the near future in Asia we won't let across government there were very very border multidisciplinary as 1 unique characteristics of the 80 people in danger we got people work on defense and security matters away free health care Education innovation policy criminal justice and that multidisciplinary makes enjoys quite a lot about our values
The way reaching the fundamentally where we are provided evidence In the Anglo-Saxon world The concept of evidence-based policy-making has had a long tenure and has been on the rise Doesn't seem to be disappearing And I after say what we're the guys provide evidence to support evidence-based policy-making we support sells event a number Groups Some of the policy focus on the plateau focus clearly do what I gotta be talking about today on fate since that innovation and technology policy and that is still welcome if only just give you never know where I'm going on as I said earlier I will just you the review of some and Harold research evaluations
Important measuring research pack his report mild then DDT case studies 1 is in itself has had a massive impact and you can be on the order has had less impact by things got interesting way of thinking how we're going forward in this domain and pick up some lessons and discussions so I like to think so I like I like the little ITT like that we evaluate
I'm research for reasons Advocacy accountability analysis and allocation to what I mean by this what think Every chief executive of IT research funding institution every Vice chancellor the University has little black But where she or he will be out to go to cocktail parties dinners or events and talk about the impact their agency institution is having on White a social wider society and that's good that near on not be hitting that setting thing is Rudy important part of that process as researchers we want people representing us at that in review the economy life and advocacy is really good I would argue that when Abbas at
Sees Hello when it is driven by Analysis and his representative in making a case rather than being driven by my worry too much That is driven by Some accountability is key to the match that's already listed units this room I suspect bonded free taxpayer Contributions I sometimes you may be funded by a B and research charity or philanthropy and I don't think the arguments for when that there's a lot of things As researchers As research community justice respect other areas of public life be accountable for how they spend their money has a research team we have a responsibility to be accountable for how we spend money and demonstrate that we provide value for money and spending money they should I really interested in is pretty good a analysis of what I really would like to understand is what will search for nothing is slightly I wanna Quinn Utica other areas Educational health care much better understanding of what works what doesn't work the limit of science and research we have very little understanding what works and doesn't work on a given what sorts researches Obama might about standing up study paradoxical but to illustrate that we don't really know It's best to fund Research from individuals Projects we don't really know rivers bears have readers institutes will have some research and teaching near there is show format but it's pretty inconclusive it's pretty small numbers and is relaxing empirical basis on the whole give your exceptions but I think it's quite a week's rest to other things driving My group ran is Trudy tried to develop by almost an algorithm we we actually have a formula we put up to scare people know was not a full folks is people's mind renewed the top 4 minutes and his research funding should look this is really start on the stand some of those factors that lead to recent success not touch left very briefly and then the I'm full Fuentes allocation which In EU but nowhere else as far as I know at this time is really got up the agenda in the last 5 years we just introduced a new system for research excellence framework rest which is his kicked off and it's going to be universities have submit this framework Next year and 20 % of university funding is going to be determined by the impact those universities have And as far as I'm aware that the the point that Impact has been used on that scale to joint research and that in itself is a whole new set dynamics between universities universities refocusing their missions around impact because we're talking we're talking here so that over a billion pounds research funding over 5 years more numbers here this is really a quarter Tension says the wife of the head then you we have a number of tools to try to evaluate and measure research thing the key thing is that no 1 who is perfect any sensible research evaluation of impact assessment will use multiple tools and I think and hope by illustrate that in the developing somebody's methods that key to improving the way we understand the return on research invest want to benchmarking compare yourself rather like institutions not the concept context league tables we see too many tables ranking universities near that that's it he asked what we really want to do is to drink universities and then have a discussion about why is it that I'm No. 2 in No. 3 what you doing well what could I improved it should be driving rather than being it's a team sport sometimes University rankings almost like the a European Championships couple That's gotta be quite unhelpful Doing so they self evident that I'm getting information from research is that what they're doing touch on the developing conceptual models logic models of understanding research process and this is quite difficult because all literature points out correctly that research is not linear but Hawaii research is funded
India had Eastern Conceptual models that allow you to conceptualize held on to the non academic research impact and have capture and illustrate what others later debut may choose is a topic I assume most recently familiar with Hajji measure citations on research papers but go beyond that and the thing is something of relevance to you had had we measure citations on other types of documents and to the Web site scope as we got back community Iran's formalized peer-reviewed publications Could you outside
But increasingly People interested in making citations all clinical guidelines citations in policy comes ahead we captured that systematic way really undertake a detailed case studies I think All really what that come from Quantitative and if need be said that is not unimportant so we do a lot of case studies of case studies in this theory Rudy allow you to get city to the texture of the research process issues as serendipity those issues of culture and values just so so important successful research seemed very Cesare research from refund when we're examining them for facts that analysis and case studies tend to be the most powerful tool we have Economic analysis on injured in example that I am sorry for that too much and then review which again and everybody in this room familiar with vast appears thing is as high-quality research teaching is as high since as we have the tools they need to be improved they need to be integrated but we have a a toolkit say he could not consider those the white Hello I'm questions in a matrix end point here is that Different Approaches have Different requirements for most of the toolkit logical perspective and what I'm going to do because of time constraints And because I want to drop There are many good sought to those days I will briefly touched on the other to raise revenue also discussed very happy to do so suddenly just give you um Mikey messages I repeat the end the summer key Mr. early on in case you need a or whatever and I like to stress that people that you really need to understand why
Your mission what is your objectives said he is sitting with a research fund and they invited us in Evaluate research program we spend a lot of time getting 3 because that would try methodology said if they want something which is going to be evidence given At It's a very very different approach than if they want to you Allocate research funds and its end Rudy understanding from the motivational research impact assessment is key
That doesn't mean that I am usually squeezed because it is a massive kind ranking those for a point at River where where there are using multi method multidimensional approach just illustrated a series of methods multidimensional flight acts You put on me but a multidimensional that theories academic impact we should therefore be looking at their impact but this will not impact don't spend enough time looking at this impact on people's careers training Ph.D. post docs impact on publications and knowledge production this impacts of helps us we Knew what when Eddie going to an evaluation think that different dimensions of impacts on the Net multidimensional Archer is useful thing is that say acknowledged that is not easy measuring research impact I don't think anybody's got upset but at the same time you can't say it's too difficult and require a two-state that's a legitimate argument
And I would argue and have argued that we need to move away from the . isn't doing this Iran's advocacy where senior leadership in the research community on making the case for research on Don't be little but we need to move away from that the U.S. too often the virus this 1 where it's about accountability about where researchers really demonstrates a society that bringing and the benefits that I think is really strong so you near this whole debate sin in other formats the issues of public understanding There was heavy on a thank you people you people
As in the known researches have a better understanding of the research process have complicated researches and not predicted it is 70 capable of the accidents where comes on the easier it is to generate generals Porter research rather very determined stick approach that kind of applied to the top of the standings says like message is that we move on to my 1st Some exemplar And this is a report we did Collaboratively with free groups in you can publish about 4 years ago The other 2 bits healthy research paper breeder University and the Office of how feature bombings which focuses on private sector R&D funding They had rented this study is that there a series Reports coming out those in U.S. then in Australia in 2003 that looked to be Croton Croats economic returns the by medical and health research and these Um returns will make sure You see the numbers on their side said I was investing in my pension fund still these numbers out the warmer money in biomedical research I have that's what you do pct and we disagree with Very critical of the methodology by further studies and I'll come back explained in more depth and we're delighted with the UK a cattle medical science to report in 2006 that said research founders just for research to assess the economic impact on medical research so remote embarks on his 1st Were they have research Bruno and myself City of team to do this Went to the Amex tender which we Bid for what which is cut is scary when you been criticizing other people's work and then you have the opportunity to show have become a set itself up but There is good fun so what what do we do and this is this is the study 1 slight rich conceptualize what we did and the detail behind this is it It is phenomenal effectively what we need To do with estimate for bits of information and we need to do estimator much money spent on research we need to understand owned by Medco Health Research we need to understand what the hell getting is phone that research we need to understand what each comics is from that research site I had no idea DEC uses the case but maybe sore spot and that may be Spillover from having A university situated here so that's generating additional economic benefit which was not a direct objective but they are funding came we different that's very clear that Iran's which we have a number of biotech companies and that spun out The university end generating gnomic well That area and onto me to estimate what I libraries and crucially and 1 of the things that they Australian and U.S. study forgot to do was To understand what the time lag between input and output dropped from its their studies they assumed it was instantaneous argued that Emsworth research funding and you give me some help games in each of us nonsense But as state and so are creature for data points so we focused in old codger vascular and mental health research As Asturias Pat Richards cardiovascular 2 reasons done a lot of working cardiovascular what we had a good understanding of research developments in the history of cardiovascular and 2nd cardiovascular is pretty good quality and And dimly understood that we chose mental health because we really wanted to stress test all model because the mental health wasn't like that yeah if you look at me Helping to mental health that pretty pour Nunavut investments and mental health is mediocre and the data is used the definitional issues and slack So we didn't have to stress test effects Encarta vast but just to illustrate back to rebuild up this time series of cardiovascular research spent and and because 1975 as we knew we had a go back a long way this time thank you she did the time that was Data And 1975 was a good break point on 1 of the year and the the MRC the Medical Research Council data series because they shifted data definitions and we Batchelder dataset by backs EU 1917 a lot of research compiling that series But in terms of the modeling we could stop I have to say Yeah she's speaking to an audience that yourselves on that so data archiving this information that we really had a researcher spent 2 weeks in the archives in the Research Council end Extract him out And this is 1975 is over You're just be leave at this information the shortfall would be great sets and end the different lines here for less interest but also different from those fund basket research into the report we get into law detailed how we estimated at any assumptions may cause loss of assumptions
Bridges above all best Estimates as regards was between 1975 to you can make public in Chapel funding was arranged to brilliant The 2nd issue was then we need to dance and health care 1st issue which is not a critique of the US Australia study is weakened the Kabul how things went fingered just the Improvement in cardiovascular morbidity mortality over a period of time but we're looking on ensuring the health gains who actually driven by research so you or some other self gangs are more social living better housing people may be more exercise eating better whatever issues so we were lucky in the cardiovascular case Wilson the mental-health case amid a spectator saying some oncology and we don't have a cardiovascular case there was a previous study which identified the most important research base interventions that led to help gains in cardiovascular disease and then we quantified those using an indicator core qualities which is a quality adjusted life here on which basically is a is an indicator used by health economist is pretty accepted and health economics field but does not economist inmates A slightly comfortable because what it's saying it is ahem any additional years of quality life you get Fulham stopping smoking is as the simplest answer or take that Or whatever and we did a massive hitched review compiled a Data to estimate the gangs as you can see here we end monitrice quality gains by saying the each quality was worth 25 thousand pairs never again you say why 25 thousand barrels and that the number
The National Health Service Users in the UK determine whether a treatment is cost-effective and again when the differences between all study and other studies and U.S. Australia was that they use a pressure me that we did a whole are sensitive to announce is changing fast and other bits but we have to . 8 million qualities quality just like years we gave a value of each of those 25 thousand parents You get 69 billion parents and then crucially And there's a cost of living off testing new innovation is the cost new pharmaceuticals I usually most events and generics so we had to estimate what the costs were we estimate the cost to be 16 billion parents say we estimated health game worth 50 free be impair compared investment Of to be impaired we don't we haven't double time except get too excited but is starting to get a feel yeah we're talking between tells there's a big numbers we had big confidence intervals around this in sensitivity analysis As well so that we get on to our look at the time that we estimated time next 17 years and we did that by looking at 3 Cycle time as we called it between their own clinical guidelines that is what is being GM Europe all the paper cited wanted clinical guidelines compared the publication of godlike technique is used quite a lot of patents analysis of replied found at the back that That was 2 . 5 years guidelines and then we knew that there was a period between spending a publication so you you get a research grant duty-free years stopped published and there's going to be a period between the physical God being published and being implemented by the nation's so again we estimated gets make a time back 17 years and enemy reflex that's not as an aside just published a paper doing review on East Timor X and there 4 other studies reading is what
Which come up with 17 years using very very different methodologies and we which and the title of the papers theology 17 years waters the question some of you may know wearing that from to it at at 17 year and then the full of component is understand what spillover now measuring becoming spillover from an economic point of view is Rudy really challenging and we don't have the resources to do that as part of this study just bitten by a grant of a treaty that next year so we did alleged review almost Spillover effects and basically B best estimate that which which is around 50 % that No major problems that Said because was a literature is a is quite old U.S. dominated and see and agricultural research so we're making assumption that week Translate I'm not mortal She gotten so anyways we come up with a incinerator returned While 59 % which is a lot less but could again you'd best The stock market he knew vigorous and so that is a lot less than 5 which is Australian friends concluded so why why why why do we have these fundamental differences 1st thing is birth
The U.S. industry will studies took a top-down approach so they looked at full mortality morbidity really did approach that focus on research base intervention so we identified and that it should be 13 research base interventions will be very confident that the research that again and not some other social dynamic and that means we're being considered because there was this stuff they they were being whatever
And that The bitter always got me was here so that they acknowledged that not all of us On health gains to the actor Peter John and in the reported mangoes so we don't know what is written assume 50 % pneumonia that
On the basis that is already mentioned annoys reader Tyonek saying they assumed that those investments are they do not let off a health-care costs we netted off 16 billion parents Of health ends We used at a lower value statistical value of a life ban for what they did in use quality players Use the value Like no way stats statistics about life but then value was effectively free times higher and what we found is that when you um actually undertook all model and made all these assumptions where she come Roughly 500 % as well which are things quite interesting so that the date it was called the urgent and the data but it is kind of the assumptions of the methodologies I would argue we improved This approach so Let me just talked briefly about the impact that cause I think and you can be Has had a major impact Impact As a research group Weifang quite Conte looked on as people have approved the use of magic 50 % not equity meetings where people tell me the other party vastly research is good but really did you reach up to write a book cab gets you sensitivity analysis and put it had an impact of having had an impact Was a number of of years and that works in the political and that's not quite understand that it has an impact think it was reversed on unknown say that if you read the report every assumption that every bit of data is in there I think that's really crucial so people could go in the former Chief Executive comedic Research Council when the peak commissioned jobs came University Cup will not go yet she was saying the best thing about that is that the money disagree you under white disagree think again in this space Policy when people don't advocacy is too easy to divide by 100 % Add to what I really think about what you're doing and the last thing that I how however the study this quotes here comes home loss on that they do with it So in the UK we had a spending review in the new coalition government and that spending review the political spoke of spending in all areas of education and research and the 1 area that was protected by medical help foreign that I would argue part it is because the study but also next on suggest that we just take it center that saving rather than the contract is very well for And then the other thing which has a research group which really weak I was described here which was in an editorial in nature and acts referring to study Without my grades that work so that that's
Become evidence a very different approach to measuring researching the relief Interesting enmeshed emotional viewpoint difficult challenging lack of data developing methods developing the conceptual framework and I leave Revisited and helps UK cites a lot but also Dubinin agenda on how we deal is better that any move onto something site differ where they we being as successful and discuss why some thoughts I I I think is an interesting approach when tried stand research impacts again appealed academic impacts and social economic soft and this is pleased to work with the number of years ago are fighters research campaigners was put off by you carry medical charity in UK Katie funds are Vargis research but a much of the basic things like types of all the way to the the more clinical might make the bench debates to say they are the 4th largest Under the trapped under new among the largest Kunduz off artist research in Europe at that time spent about 25 million pounds a year of his research and that money is what would allow money is raised their nations and legacy and they we read We had done Seeding studied them which is case study based on which they can play help very but she his time came just as sitting area Case studies but We can't see case studies on every single project images can't us going much is anyway you know Croatian rises thinking and come up with a way which is going on to capture and measure Our impact but is relatively cheap and to feasible for the operational perspective and the other thing which we had identified them but they were well aware of The fund as best as they were quiet bag on told us summit Intergraph reports that you have submitted and pulled under the pointers Research Fund is that gasses complete waste of time and effort by from the research perspective and research Bob And so they were thinking of saying basically coming up with a deal with a research communities and we're not meant to require you to you Do you and got reports anymore but what we do want to do Is former you ever appeared as time because as we know that the At take more of that period of when you walk into got report which is usually within a year circuit targets they cannot do they research community with effective city you will get follow-on funding from last year's battle sources at 1 year 18 months for ideas and potentially beyond legacy of after 5 years under several research and there so we did is we we works with equal citizens with 40 Opry searches to the case that research and try to break up into a wet base questionnaire which can be completed within 30 minutes that was exclusively yes no questions abound
And basic Peter Foster had you had a impact as you working such token godlike half you done any work and local school taught by a research at local school had Brief your local post along the border Chevy appeared in media and to read yes yes there says was quick to fill time and And those 187 questions came from a steady work I work working 40 researches we've never taken a 187 opted to 120 as every time we do more work reacts that the point is that we know that when people but surveyed the vast majority of them like a 70 per cent think it is pity within 30 minutes just We're paying for pressure to what we did use them we broke down these types of research impacts these 5 categories which is based on something called payback framework which Colleagues have become to research develop saying things those papers like and they identified a slightly different types of research impact on 1st to July car Academic impacts and the last freer more white impacts and society And social change so we we identify days we took place categories to do it in those 108 7 questions as much as 400 questions of troops onto till I focus on research targeting and capacity building we can use it
Drop into that and then there's another set of quite simple questions here to delete the funding was a interactions industry did it develop any research tools on digital track of academia Drop into that Had an issue discussions average for Knowledge Exchange a guest let these findings he took Co application for funding yes somewhere successful yes again you getting illustrations array of new circuit rooted in the on Cheney's questions so you don't necessarily also 197 which again is So we and their people to see also the begetter a color-coded should become self-evident more you weekend Entergy take a gaunt and Mr. buildup over grants and build up this week and an impact right which is great
It could be really interesting point today is that whenever developing a database of multiple research grants And multiple types of research in which means that we can start calling and statistical analysis regression analysis across the state And that's going to be crucial for still kind of in development
But we forget This is useful Version of it is used by a nationals have research and you can The largest from there And applied research you can tool then gone told us now have that in this questionnaire on an annual basis gaining it updates the The Surveys remember what I said last year that to repeat what I said last year but they develop impacts of iron and he's a some very recent Bits of data on that allows you to Just describe what is going on said colors here are different types of research funding and these different types impacts here you can see if you like in total is a 100 and 80 or Ph.D. was supported by the stream Research funding talking more interesting Is unit here the small number patents Before that in their patents were being captured and probably more interesting I think is is this got here where we start to look at the impact of the tolling of different research guards
Like and this is getting bad health policy impact so this is kinda National Institutes of Health Research you won't have an impact on how policy that gotta be a mission so we knew that after 10 years of 12 % big losses having some type of impact on how much we never knew that nobody ever knew that we don't it as a good number of but is we have what number which we start comparing answered this is very much a work in progress But it is an approach we've been trying to tight where we can basically desegregate impact in a series of data and buildup data then I'll ask others to Ontario to try to understand what that difficult equation I talked back introduction could look like a repeats wants insist this too was there being used by some others the MRC defining a similar Based Only amateur we put some guys in Canada happy about if it wants assault the tool but is this conception approach to desegregate Impact studies and charge bank effects data subtlety gathers to um pieces a work at skiing can't ability just focus for a couple minutes on analysis allocation by we're going to have just 1 slight an ancient harem poaching case study researchers and
It is really useful really important for the analysis of researching that's getting to what factors Leads to recent success and what we've done Caesar studies in off right arthritis In cardiovascular research doing a big 1 narrow in mental health research is to develop case studies over a 20 year period we go back 20 years when the research funding incurred And build up its narrative about war happens that research from speaking to the people is a semi his analysis build up these narratives to a very tight structures can compare across and we could be enriched or fighters spot we have 16 case studies the cardiovascular we had better case studies which the case study researchers There is a large number as a center stage on the basis of larger case up and then what you can do is that you can we don't basically bring together Panel of peer review is to rate use case studies on different dimensions in so we quantify this case studies and then we undertake an analysis where we look at babysitter The quantified impact did time mention and compare it to the qualitative information a case studies the series stopped white and somebody factors associated play impacts and 11 so give you a couple examples to illustrate that I
Area that we increasingly find anywhere be surprising is impact collaboration soap and war teams collaborate the greater the impact of the recession again I'm talking about normal academic impacts Errico under that agreement but this is verified that from the more social perspective some the more interesting observations is that Tentatively In a recent report we don't quite have the numbers to But on money on it but this being a practice Among on research funded largely charity that they will take what the EU large national body doesn't want to be the Medical Research Council rate with gone on 10 not so use genes that sent the next tests or others could get quality and dating get handed over research chapters this is effective policy Australia when that occurs Of the policy quantity that will be no a good idea but it knew that there was a rationale said to transaction costs like there's some so I II fund area and fast 18 than the whole What's driving that is help inform research policy research And make them more effective and more efficient What the 2nd area His allocation dimension as in the UK we have the research experts framework which is going to allocate 200 me Adams year over 5 years On the impact research is having a higher education and the way that you are pan-Arab his true peer review researches Are having to write a study of the impact of research and this case studies about 45 pages the whole thing fruit to peer review committee and they will get going and they get rated on there Significant significance of every impact and the reach family people written will also work Trying run out UK universities think readers have had you identifying in institution is not as easy as it and then had to strengthen the case with undertaking critical reviews on behalf of some universities in UK which Ross great game to see all this stuff Which we would publish Charlene's really interesting if that's the case study allocation approaches Tried out of things that said experiment live and work but what I find really exciting thing again something which may be present you guys is when this is done they can't be something quite thousands of these case studies we can find a way of data mining the case that starting to tease out some of these factors associated sex that we're that's much more half actually survey tool that we usually developed which we started thinking back before referee came and nothing is a really exciting change you have to stop thinking through how you with and data mining these 5 has an impact statement which will be ranked scenery somewhat behind some of those will be made some and then seem Deacon identify common factors associated savor the high impact case studies
Humanity's will history a subtle matches biomedical sciences about Ref Ross is that we're being placed Other disciplines and it's great having a discussion with stolen about their impact because they come in things have impact that she would stop area discussion with people that you have impact stopping people think about the having soaked That said The largest we want less and discussion of a happy to take some questions and that's it just somewhat Do I free got across the research evaluation requires What tools but I think it evaluation It's about weaving this tapestry in Tokyo Specter means different tools together And not relying on a single to 0 that's kind of what makes it different causing multidisciplinary teams to see me that so to repeat my misuse ahead but got across that you need to know why you measure impact way measuring evaluation research with objective research which I hope got a course that you must take a multi dimensional MultiMate approach is not easy but is not possible as a really important point to get across and as a community I think we need to move away from advocacy position we've been in almost a generation when it comes time thinking that research impacts to 1 where Sasser science as a star understand what works and we need a practical and face the sun policymakers and I think the Sox policy community Condon Easter the told as I said in the introduction at site paradoxical research suspect proving understanding we don't really understand So mothers or be question how