Keynote: How to Fix a Scientific Culture: Psychology as a Cautionary Tale
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 43 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/38190 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Erlangen, Germany |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
4
5
6
7
9
10
11
14
28
29
32
33
34
35
38
39
41
43
00:00
Fatou-MengeBitSoftware developerContext awarenessSign (mathematics)CuboidComa BerenicesGradientStudent's t-testComputer animation
01:04
Multiplication signPoint (geometry)Content (media)Observational studyQuicksortPosition operatorProper mapRandom number generationComputer hardwareNumberRight angle
02:48
Position operatorBit rateRandom number generationMultiplication signDifferent (Kate Ryan album)Random number generationPosition operatorType theoryBit rateArmCycle (graph theory)Rule of inferenceObservational studyPreconditionerComputer animation
04:29
HypothesisDegree (graph theory)Control flowDrop (liquid)Observational studyPhysicistPreconditionerCognitionBit rateMultiplication signLine (geometry)Sound effectResultantRight anglePosition operatorP-valueSoftware testingError messageDegrees of freedom (physics and chemistry)CASE <Informatik>Latent heatGenderCondition numberMeasurementTwitterHypothesisGame controllerNoise (electronics)Data analysisTerm (mathematics)Different (Kate Ryan album)SimulationOffice suitePunched cardStatisticsThermal fluctuationsComputer animation
08:06
Data analysisData miningBitTerm (mathematics)Noise (electronics)Inheritance (object-oriented programming)Auditory maskingProcedural programmingP-valueBlogMultiplication signFitness functionDrop (liquid)Metropolitan area networkData miningArchaeological field surveyEndliche ModelltheorieArmThresholding (image processing)Arithmetic meanBridging (networking)Computer animation
09:47
Degree (graph theory)Universe (mathematics)NumberDifferent (Kate Ryan album)MeasurementEmailArchaeological field surveyResultantObservational studyNoise (electronics)Bit rateArithmetic meanComputer animation
11:03
FlagOrder of magnitudeReplication (computing)Order of magnitudeOpen setHypermediaP-valueQuicksortObservational studyScaling (geometry)Projective planeSound effectCollaborationismRandomizationThresholding (image processing)Arithmetic meanThermal fluctuationsSign (mathematics)EstimatorProduct (business)Computer animation
13:13
Table (information)NumberObservational studyArrow of timeArmEstimatorConsistencySoftware testingPoint (geometry)SoftwareCASE <Informatik>Physical systemExistential quantificationProbability density functionComputer animation
14:21
BlogStudent's t-testGradientStatisticsConsistencyResultantSoftware developerInformationHecke operatorSheaf (mathematics)ConsistencyMultiplicationGradientStudent's t-testNP-hardBlogSampling (statistics)Traffic reportingDifferent (Kate Ryan album)Machine visionObservational studyElectric generatorMereologyNumberWave packetGenderComputer animation
17:00
BlogStatisticsNumberStatisticsDistribution (mathematics)Machine visionMaxima and minimaInheritance (object-oriented programming)QuicksortObservational studyPosition operatorService (economics)Computer animation
18:05
HypothesisObservational studyControl flowStatisticsObservational studyGoodness of fitNumberSoftware developerCirclePower (physics)ResultantTheoryInteractive televisionGame controllerPoint (geometry)HypothesisSound effect40 (number)Condition numberArmNatural numberFrame problemGroup actionWater vaporSign (mathematics)Coefficient of determinationCausalityLie groupComputer animation
20:46
Decision theoryMultiplication signTimestampLevel (video gaming)Type theoryObservational studyOnline helpTraffic reportingComputer animation
21:35
Software developerExplosionIdeal (ethics)Peer-to-peerLevel (video gaming)FeedbackArmFile formatResultantSheaf (mathematics)Observational studyTraffic reportingLevel (video gaming)Decision theoryComputer animation
22:46
Mathematical analysisData modelSoftware-defined radioSoftware development kitDecision theoryEndliche ModelltheorieDifferent (Kate Ryan album)Latent heatSound effectInteractive televisionCurveQuicksortBit1 (number)Order (biology)Thresholding (image processing)Degrees of freedom (physics and chemistry)FamilyData analysisMathematical analysisContent (media)Shared memoryLink (knot theory)Group action
24:43
Open setSoftwareMathematical analysisScripting languageBitShared memoryResultantSound effectMaxima and minimaScripting languageDescriptive statisticsMathematical analysisNormal (geometry)SoftwareOpen setComputer programmingSoftware frameworkProjective planeWikiFunctional (mathematics)Sign (mathematics)Computer animation
26:18
Peer-to-peerForm (programming)Open setServer (computing)Software frameworkFunctional (mathematics)Revision controlLink (knot theory)Scripting languageTimestampProjective planeWikiGame controllerComputer animation
27:33
Open setLength of stayMaterialization (paranormal)Observational studyWeb browserOpen setGame theoryArmLine (geometry)MereologyMultiplication signDirection (geometry)StapeldateiNumberCartesian coordinate systemTerm (mathematics)Matching (graph theory)PressureComputer animationEngineering drawingDiagram
28:43
Reading (process)Text editorAuthorizationShared memoryOpen setPeer-to-peerMathematical analysisMultiplication signDecision theoryArmJSONXMLUML
30:19
Hacker (term)Natural numberBounded variationNP-hardObservational studyFilm editingMathematicsSoftware developerIterationSimulationType theoryPreconditionerNumberLimit (category theory)MeasurementFaculty (division)Selectivity (electronic)Replication (computing)Physical systemBounded variationDivisorPosition operatorCartesian coordinate systemInstance (computer science)Multiplication signSign (mathematics)Computer animation
34:09
TelecommunicationPOKEResultantCross-correlationProduct (business)Inheritance (object-oriented programming)Instance (computer science)NumberDivisorObservational studyVideo gamePoint (geometry)Coefficient of determinationJames Waddell Alexander IIOcean currentComputer animation
35:56
SineOverhead (computing)Sound effectMultiplication signPhysical systemOcean currentEndliche ModelltheorieCartesian coordinate systemTwitterView (database)Exterior algebraComputer animation
37:11
System programmingExterior algebraPhysical systemGoodness of fitComputer animation
38:00
System programmingDigital filterObservational studyHypothesisExpert systemChecklistSet (mathematics)Cartesian coordinate systemStatisticsAssociative propertyMessage passingDirection (geometry)Arithmetic meanProcess (computing)Power (physics)Automatic differentiationReplication (computing)ArmRule of inferenceMultiplication signSelf-organizationLevel (video gaming)CausalityComputer animation
43:05
MereologyMoment (mathematics)Metric systemMixed realityProper mapRight angleStatisticsBitNatural numberTerm (mathematics)Observational studyEntire functionSoftware developerPropositional formulaGame theoryCartesian coordinate systemStandard deviationPoint (geometry)FreewareGroup actionAsymptotic analysisPhysical systemCASE <Informatik>AreaPosition operatorQuicksortLecture/Conference
47:56
System programmingStandard deviationSimilarity (geometry)Type theoryMetric systemGame theoryCausalitySign (mathematics)Set (mathematics)BitArmMathematicianNeuroinformatikOcean currentMultiplication signBranch (computer science)Film editingContent (media)PhysicalismProjective planeUniverse (mathematics)Cartesian coordinate systemParsingPhysical systemScripting languageProper mapPhysicistNumberHypothesisMetreSource codeField (computer science)NP-hardArithmetic meanReplication (computing)CollaborationismOverhead (computing)Nichtlineares GleichungssystemPoint (geometry)Message passingInheritance (object-oriented programming)Chaos (cosmogony)Probability density functionRight angleLecture/Conference
57:00
Natural numberGraphical user interfaceDifferent (Kate Ryan album)Sign (mathematics)Universe (mathematics)Data storage device1 (number)Process (computing)Drop (liquid)NumberEndliche ModelltheorieContext awarenessMultiplication signRight angleArithmetic meanPhysicistGame theoryPhysical systemTheoryArmSoftware frameworkPhysicalismStatisticsOpen setNormal (geometry)MereologyField (computer science)Computer animationDiagramLecture/Conference
01:01:03
Computer animation
Transcript: English(auto-generated)
00:06
Okay, hi everyone. Good morning. I feel a bit like a mole. I'm pretty sure I'm the least techy person in this room. So my name is Julia Rora. I'm currently a grad student at the Max Planck Institute for Human Development in Berlin, and so I am a personality psychologist.
00:24
And so today I'm going to talk a bit about how we can fix a scientific culture, but that title was a bit too ambitious. So actually I'm just going to talk about recent developments in psychology. So you've probably heard about that, like, so-called reproducibility crisis, and it looks right now like
00:41
psychology is facing a lot of, like, huge issues. Like, are we even a science? I don't know. I hope we are. And so I'm going to talk about these problems, but also about possible solutions in the wider context of how we do science. And since I assume that most of you are somehow involved in science, this might be, this is hopefully irrelevant for you as well.
01:03
So I'm a psychologist and we do experiments. So I want you to imagine you're now a participant in this experiment, and you see two curtains. And behind one of those curtains, there's a picture of a blank wall, and behind the other one there is an actual picture. And now you have to guess behind which picture, behind which curtain you can find that picture.
01:23
So you might go for left, and oh no, it's the blank wall. That's a miss. And so we try again, and this time it's a hit. Oh, yeah, we found the picture, and that's great. We get one point. And so this is the basic setup, and now there is something special about that experiment. So the first thing is, some of the pictures are special.
01:42
So we might choose the left curtain, and there is a picture with an erotic content. And so these are not the pictures from the actual study, because I think the pictures were fairly explicit, but I wanted to keep this safe for work. And so the one thing is, some of the pictures are special.
02:00
But the other thing is, so how should you know where the picture is? Well, it's actually randomized with a hardware-based random number generator, and so there's no way you could know where the picture is. And now the research are doing the study is interested in how many times you guess the position correctly. And it's kind of like, well, why would you need to run that study? Like, we know it should be 50%, right?
02:22
It's just like a random number. You cannot guess the random number. If everything is implemented properly. And so the only possibility that you would achieve more than 50% would be like, some sort of special precognitive skills. Maybe you can't see the future. And now the astonishing thing is, so this is a study that was published in one of our flagship journals, JPSP, by an eminent social
02:46
psychologist, Darryl Bam, and that's exactly what he found. So across all 100 sessions, participants correctly identified the future position of the erotic pictures significantly more frequently than the 50% hit rate expected by chance. So they got it right 53% of the
03:04
times. And it perfectly makes sense that it only works for the erotic pictures because it's an evolutionary advantage to be able to see the mating opportunities. And so this is like how that's interesting. I mean, well,
03:22
I wouldn't rule that out. Actually, I would rule out that this works, but extraordinary claims require extraordinary evidence. And that's what Darryl Bam provided. So it wasn't just one experiment, it was nine experiments. And it wasn't just like small experiments, but all in all there were 1000 participants involved. And so now you could say, okay,
03:41
so most likely this is something about the random number generator, right? Something must be broken. But he was fairly like rigorous about this, and I think the experiments, they took a long time. And so this is like probably the most bizarre thing I've ever read in a psychology journal article is like he has that nice,
04:01
so he distinguishes between like precognition and clairvoyance and psychokinesis, and how these things relate to different types of random number generators, and what we would predict based on these things. And so this was published in 2011, and I know that quite a few people were like, huh,
04:21
what do we make out of that? It's our most prestigious journal, at least, in social and personality psychology, and they published this. And so there's a study following the best practices published in a flagship journal, and it shows that precognition is real. Well, one possibility is like, okay, precognition is real. We just have to accept that it's significant. And the other thing is,
04:44
well, maybe precognition is not real, and personally I have a lot of trust in the work of physicists, so I would rather go for maybe psychologists are wrong here, and maybe there might be something wrong with our best practices. And so the question is, what exactly is wrong about our practices? And to understand that you have to understand how
05:04
psychologists do statistics, and it's a very simple thing. That is, for example, we observe an effect in the data. We observe that participants get the picture right in 53% of the cases. And now we compare that to a null hypothesis, and the null hypothesis is normally there's no effect.
05:22
So in this case, the null hypothesis would be, it's 50% actually in the population, and so this like 53% might just be random fluctuation, just like random noise. And so what we do is like, we calculate how likely it would have been to observe that effect, given that the null is true. And
05:43
if that turns out to be more than 5%, then we say our p-value is not significant. And so we do not reject null hypothesis, because the data could have occurred under the null in more than 5% of the cases. But if the p-value is below these 5%, well, then it's significant. And so we reject the null hypothesis of no effect, and we establish the effect exists.
06:05
And so this is not exactly sophisticated, but if you do that properly, so you run a study on an effect, and you test it, and you do it again and again and again, then what you ensure is that the long-term error rates are controlled. And in this case you will ensure that if you investigate something, and there's actually nothing going on,
06:23
it's all just noise, then you only wrongly conclude that there is an effect in 5% of the cases. Which is nice, so this is long-term error control. The thing is, this is not how psychologists run their studies, but, and you probably all know that when you run a study or analyze data, there are some, like, degrees of freedom.
06:41
And so, for example, psychologists might decide to include more than just one outcome measure. So, for example, if you use a questionnaire for personality, well, there are different ways to measure personality, and so you might just include a bunch of questionnaires. You might also just, like, study your experiment, and then you see the trend you predicted. Oh, but it's not yet significant, so you just collect a few more participants and test again.
07:05
And there are fairly innocent things, like, oh, maybe this is confounded by gender, so we have to statistically control for gender effects. Or maybe it's just, like, you tried a few more things, but you see that some conditions in your experiment just don't work, so you just drop them and never talk about them again.
07:21
And then, of course, you can combine all of the above. But what does that to our false positive rate? So the false positive rate could be somewhere between, like, zero percent. We never have a false positive if there's no effect, and 100 percent. Every time there is an effect, we will always conclude, no effect, we will conclude there is one effect, which would be fairly bad. And so this bright blue line is the five percent
07:43
we're aiming for. And now, each of these practices results in slightly more than those five percent, actually, considerably more, so we go up to 10 percent of false positives. And now, the scaring thing about that is, if you combine all of the above, you can get up to 60 percent in that specific simulation, and
08:02
I'm fairly certain you can get it up even higher if you try really hard. So that kind of means, like, with a creative approach to data analysis, you can get anything significant if you want to. And there's actually researchers in psychology who just know that and are, like, boasting, I can get anything significant. Although, don't tell anyone that I told you that. So,
08:23
you probably know that, but psychologists kind of had to learn that this is what's happening, and we coined a term for that, that is p-hacking, because you're hacking your p-values till it's below that threshold, though I think some people don't like the term because it sounds like it's super intentional, and it's probably not that intentional.
08:41
Also, I think psychologists don't quite know what hacking means, so they think it's, like, evil people sitting in front of the PC with maybe wearing a mask or something like that. So, and we have some other terms we are using, for example, questionable research practices, QRPs, which is a bit broader, just like the name says.
09:01
And there was one blog post to use the term noise fitting, which, noise mining, which I like a lot, because it's, like, going through the noise and see whether you can find something that looks precious. And one term that was used by Talia Connie that I like a lot, because it's just, like, very precise, is procedural overfitting, because this is what is happening.
09:21
We are overfitting our data, but it's not a problem with our model, it's the problem with the procedure that we just try many models, and throw in more variables, and then stop when something is statistically significant. So, we kind of now know that this procedural overfitting is pretty problematic, because you could kind of find, like, anything you want. But the question is, do psychologists really work that way?
09:44
I mean, maybe it's just a few bad apples. So this is an empirical question, and there was actually, like, a mail survey of 2000 psychologists at major US universities. And so what they basically did, they asked them for a number of different behaviors, such as failing to report all dependent measures,
10:03
but even stuff like rounding down p-values, which I find fairly bad, up to falsifying data. And now you can imagine it's not that easy to ask people, oh, hey, are you failing to report all dependent measures? So they did different things to estimate the prevalence of these behaviors.
10:20
But we're actually going to stick with the self-admission rate, which is, like, the darkest bar, because actually people just admit to it. So what you can see here is that more than 60% say, oh, yeah, I failed to report all dependent measures at least once. And for example, some people, like, more than 20% say, oh, yeah, I stopped data collection after achieving the desired result.
10:45
And selectively reporting studies that worked, which this is also fairly prevalent, so you see people even self-admit that they are doing these things. So this is kind of troubling, because we know that if you do these things, you can basically find anything in noise. And so what does it mean for our findings?
11:02
What does it mean for our literature? And so the question is, is it really that bad, though? And so there was a large scale effort to answer that question. And this is really like an awesome project by the Open Science Collaboration. So they tried to replicate 100 studies published in some of our major journals just to estimate how reproducible is our research.
11:23
And now there are different ways to assess whether these replications were successful or not. We can look at the p-values because it's kind of so important in psychology. And so what you see here are the p-values of the original studies. And as you might imagine, they are all below, like, almost all below our significance threshold because we only publish stuff that we consider significant.
11:46
But now if you look at the replication studies, they are actually all over the place. And so this is kind of bad. So we had like 97 percent of original studies that reported significant effect, but only 36 percent of the replications.
12:05
And of course, it's not just the p-value. You can look at other things as well to summarize your evidence. And so, for example, if you look at the mean effect size of the replications, they were like half the magnitude of the original mean effect sizes. So even if there is something and the study just failed to detect it, it would be considerably smaller.
12:26
And now there was a lot of arguing about this open science collaboration, and somebody claimed that the data is perfectly compatible with the viewpoint that everything is replicable in psychology, and this is just like random fluctuation, whatever, but it's actually not limited to that one large scale effort.
12:44
There are many popular findings that we're not able to replicate, including some things that got a lot of media exposure thanks to TED Talks. And so this is fairly bad because we would assume that proper science replicates in some way.
13:00
And though there are people in psychology arguing, well, you cannot step into the same river twice. If that's true, I'm fine with that. But it would mean that we should stop giving policy advice or any sort of advice to anybody. But actually, there's even more going on. So you don't even have to try to replicate a study to figure out that something's really off. You can just actually look at the studies.
13:22
And so there's the software, Stat-Check, and you can just feed it a PDF of a manuscript that is formatted in APA style, which is what everybody in psychology does. And it will just like extract the significance tests and check whether the numbers are consistent. So do the numbers you report even add up?
13:43
And so there have been different attempts to estimate how often numbers do not hold up. So how many articles have inconsistencies? And what you can see here, it kind of converges to 50 percent. So at least half of our papers have like statistical inconsistencies. And so that's just like nobody's looking at the data at that point.
14:02
It's really just looking at the numbers in the papers. So that is kind of really sloppy. But maybe, I mean, maybe it's just like copy and paste errors, whatever. So maybe there's nothing problematic about that. But actually, there seems to be like a system to that. And there are some fairly prominent cases where just nothing holds up.
14:21
And so one example for that is the Cornell Food and Brand Lab that is run by Brian Wansink. And they have those really nice infographics, really catchy, like food TV watchers are 11 pounds heavier. Oh, but only those who cook a lot. You can probably already imagine how that result was found. But anyway, Brian Wansink did a lot of work on consumer behavior and food marketing.
14:44
He wrote a book on, I don't know, either mindless or mindful eating, one of the two. He had a White House appointment as the executive director of the USDA Center for Nutrition Policy. And he oversaw the development of the 2010 dietary guidelines. So you can be fairly sure, wow, that's that's an influential scientist, right?
15:04
So he must do some pretty fine science there. And so there's this kind of tragic story. It all started with a blog post called The Grad Student Who Never Said No, which is already like a really bad title for a blog post. Don't do that. And so he describes how a grad student visits his lab and he kind of explains how he got her to p-hack the hell out of their data
15:27
and then how she got four publications out of that and how this is like a really good approach and how this is how you become a successful scientist. So people in the comments section of the blog post were like, wait, wait, this is satire, right? Like you're joking here, right?
15:40
But he was serious about that being like the best approach to do productive science. And so that is kind of troubling. So three researchers went through those four papers and wrote a preprint on that. I don't know whether that's accepted yet. It's called statistical heartburn, what it was called like that. And so they found 150 numerical inconsistencies in those four papers.
16:04
And I don't even know whether I could manage to get that many inconsistencies into four papers. And now one could say, OK, maybe that's just like maybe that's a problem with that grad students. I mean, maybe her training wasn't that good, whatever. But actually, it seems to be something about Brian Wansink because people now started to look at the rest of his publications.
16:25
And so there are errors in at least 45 of its publications and they are cited quite a bit. And so some of the things are really very minor. So it's things like, oh, yeah, the gender ratio report is not possible with that sample size, whatever.
16:40
Just like, OK, maybe you excluded participants without reporting it. But there are also some major issues like he reports exactly the same numbers, but describes a completely different study. And so how could that ever happen that you get the same numbers in a completely different study? At least in psychology, that normally does not happen. And my favorite one is like, so he had a study that was about like cool rebranding of school foods.
17:05
And so they rebranded carrots as like, I don't know, like x-ray vision carrots. And supposedly that increased the numbers of carrots that were being consumed. And so somebody, James Heather, just looked at the numbers.
17:20
And you just have like the summary stats and try to come up with a distribution of numbers that could generate the summary stats. And so he tried to figure out like the maximum number of carrots taken. And so the numbers in that study are kind of just possible if one of the children had like around 60 carrots for lunch.
17:42
And so it was probably baby carrots, but it's still a lot of carrots for a school child. And so I think the problem here is not that it seems like he either does not know how to work with the numbers or that he does not care enough. But the troubling thing is that he got into that position where he's running his own lab and giving policy advice.
18:01
So it's some sort of collective failure. And so wait, there's even more. So even if you get the numbers right and even if you don't p-hack your data, there are actually a lot of problems with many things we do. And so there is this beautiful syllabus by Sandra Srivastava that is called Everything is Fucked, the Syllabus.
18:22
And it's actually a pretty good method syllabus, so you can use that for a journal club. But it kind of ends with instructions for finals week, so wear black and bring a number two pencil. And this kind of summarizes how many people now think about psychology. Oh, wow, this is really bad. Yeah, well, everything is fucked.
18:40
Our science is broken and this is our darkest hour. But I actually think it's probably not the darkest hour because all these developments over the course of the last maybe six years have led to a lot of like scientific introspection. And our people are actually thinking really hard. How are we doing science? How did we get here and why are we doing that way? And how can we fix it?
19:02
So there's this idealized circle of how science is supposed to work. You generate your hypothesis. You design a study. You conduct the study and collect the data, then you analyze, then you interpret the results and then you either publish that or if it's still ambiguous, you conduct the next experiment to figure out what's going on.
19:22
And so we are failing at many points here. We fail to control for bias in the very beginning. Then we design studies in a way that even if there was an effect, we could never find it because we have such low statistical power because we are running studies with like 40 participants across three conditions or something like that. So there's very poor quality control when we run those studies.
19:44
So nobody knows whether we really properly did what we claim we did. And then people just p-hack the hell out of their data. And then p-hacking goes very well with harking. So harking is hypothesizing after the results are known. So you p-hack till you find something in your data and then you come up with a nice post hoc story why that would have been predicted from a theory.
20:08
And then you reframe your whole paper in a way that it sounds like you exactly predicted that weird triple interaction because of evolutionary science, whatever.
20:20
And then you just have a nice confirmatory paper. And so to make this all worse, there's publication bias on top of that. So it's kind of if you don't do that, you probably don't get published anyway. So the negative evidence or the lack of evidence never gets out there anyway. And so this is pretty dark. It means there are many things going wrong, but that also means that there are many ways we can try to fix that.
20:42
And I'm going to talk about some ideas that have been brought up to fix these issues. So one very large problem is that what Loken and Gellman call the garden of forking paths. So you have data, but there's so many ways to analyze it and there are so many decisions you can make and some of them are really arbitrary. So one solution to that is constrain yourself to walk only one of the paths so you don't get led astray and get lost in the woods.
21:08
And so that idea is preregistration and it's exactly what the name says. So it's a timestamp document describing what you're going to do and you do that before you run your study. So you make it public. I'm going to run the study.
21:21
I'm going to collect X participants and I'm going to analyze it this way and I predict this and that. And that makes a lot of sense and you can even take that to the next level and then you have registered reports. So this is a new type of publication and it works like this. You develop an idea, you design the study, you write an introduction section and a method section and then this is peer reviewed.
21:44
So before you've collected the data, you will get feedback, you can adjust it. And if it passes this stage one peer review, your paper is accepted in principle. So no matter what you find, this will get published. And then you collect the data, you write a report and there's a second stage peer review just to make sure.
22:03
I mean, you could still kind of totally twist the story. So just to make sure that you actually stuck with it and then you publish the report in the end. And I think this makes a lot of sense, in particular in experimental research, because it avoids all that problems with p-hacking and harking but also kind of fixes publication bias because no matter what you find, it will get published.
22:26
And I think just last week, the first medical journals started accepting these and there are already many journals in psychology who are now accepting these as a new format. And there's actually a social psychology journal, I think it's called Comprehensive Results in Social Psychology, and they only do registered reports now.
22:46
And so that is one solution to deal with that garden of forking paths. And there's another one. So, for example, I don't do experimental work. I analyze data that is already existing and there are so many ways to analyze this data and sometimes it's really just arbitrary decisions.
23:03
So another way is what you can do is like walk all the paths and then transparently report what you did. So Gelman has called that idea like a multiverse analysis. And so you kind of try everything that seems reasonable. You see how different model specifications affect your conclusions and then you try to integrate that knowledge.
23:23
And so, for example, we tried that. So I do research on birth order effects of personality. And there's so many ways that people analyze these birth order effects that it kind of seems arbitrary and people come up with weird three way interactions. So maybe people are using these degrees of freedom.
23:41
And so you can't really see it. But what we did here is we tried all sorts of specifications. So, for example, some people say you have to exclude sipships where the age gaps are too large because of the age gaps are too large. It's not like normal siblings. So we tried that. We also looked at sipships of different sizes. We only counted like full biological siblings or also put in like social siblings and so on.
24:04
And we just kind of tried everything we ran between family analysis, within family analysis. And then what you just see is like we ran a lot of analyses. The red ones are the ones that like meet the conventional significance threshold. And so, for example, you could see, oh, there's not much happening here. So most of the specification gives you no significant effects.
24:24
And because psychologists really like p-values, you can actually calculate one p-value for that whole curve. And so that probably helps if you want to communicate with psychologists. So like these are potential solutions to solve those problems with the data analysis and psychology.
24:42
But I think we have more issues that go a bit deeper. And so psychology really is a bit like a secret science. So people don't exactly describe what they did. People don't share their methods. People don't share the data. And that causes a lot of issues in many situations.
25:00
And people even feel like, oh, why do you want to see my data? Don't you trust me? I mean, you can just believe me that these results are true. And it's just like this is not how science can operate. So the easy solution to that is, of course, to open up. So we need to get people to share their data, but also share their methods. And that includes, like, for example, the stimuli, like the picture you use in experiments,
25:23
also the questionnaires you develop and also the software you're developing or the experiments you programmed. And that has like, if people started to share these, that would have the nice side effect that we don't have to reprogram the wheel over and over again, because many people are running the same experiments, but everybody is programming it themselves.
25:44
And I think this is like the bare minimum. So people should share their analysis scripts, even if they don't share their data, because like having a script is a much more precise description of your analysis than just verbalizing. Oh, I did a regression. And then you probably omit what exactly you did.
26:02
And you can't do that with an analysis script. So if we want people to do that, we have to make it kind of easier for them, because psychologists are not the most techy people. So I know that there are some people who are able to use GitHub, but it's certainly not the norm. And so one way that is very easy actually right now is the Open Science framework
26:24
that has been developed by the Center for Open Science. And so I just have like a copy of a project I did. And so you can upload all your data there. You can upload your scripts. You even have like a small wiki function where you can confess that you actually did not use Python but R instead.
26:41
And it's actually like a really nice solution also to issues with the workflow, because it also has version control, which is a pretty huge thing for psychologists, I guess. So it's kind of new for us. And it has many nice features that are somehow tailored to the needs of researchers. For example, you can keep it private. You can make it public, but you can also generate a link for blinded peer review
27:04
so that reviewers don't know who you are. And you can actually even like make your pre-registration exactly there. So you can create a timestamp document of the whole project, but also like add a form where you describe what you're going to do next. And we also now have a preprint server that also nicely works with the Open Science framework.
27:24
So this is super nice. So now it's actually easy for people to use it, but we still need to get people to use it. And how do we do that? And one answer, it seems somewhat silly, but it seems to work are badges. So I don't know what it is about badges, but my boyfriend plays those like browser games
27:43
and they just get like silly badges. And he's like, no, I still need to get my badge today. So it seems to work. And so some journals, including one of our flagships, Psychological Science, and now give badges. So if you share your data, you get the open data badge. If you share your materials, you get the open materials badge.
28:01
And I guess the best one is the pre-registered badge. Not many people get it these days, but it means that you pre-registered your study. And so it actually kind of seems to work. So what you see here on the Y axis is the number of articles that report that the data is available. And on the X axis is just time. And that red line indicates when Psychological Science introduced these badges.
28:25
And so the black line is Psychological Science. And even if you can't really see it, you will see this. So data sharing really went up after they introduced those badges. So it kind of seems to work for some weird psychological reason.
28:41
And well, sometimes you want probably to add some more nudges or psychologists don't like the term pressure, but probably also some more pressure. And one thing is the peer reviewers openness initiative. And in my opinion, that makes a lot of sense. So the reasoning is if you peer review a paper, I mean, you kind of vouch for the things that are reported in the paper to be correct.
29:01
So you actually should be able to access the data and read on the analysis. So far, I think 440 people have signed this. And now if one of the reviewers of your paper happens to be in that pro initiative, they will contact the editor and say, oh, yeah, I'm willing to review that paper, but I would need the data to do that.
29:21
Can you please contact the authors and say they should please upload the data? And there are reasons to not share data. So I think some of you are working in the industry. So, you know that there are obstacles to sharing data and that is perfectly fine. So in the pro initiative, we want people to actually declare if they cannot share the data for certain reasons.
29:43
And that's OK. You just have to make it explicit and not just like hide it away. And so this is like another idea that makes a lot of sense to me, at least. And so when we talk about these things to undergraduate students, they're always like, yeah, well, of course you should do that. Of course you should pre register.
30:00
But I thought you were doing that all of the time anyway. And isn't this how you should do science? Isn't that the rational thing to do? And so that kind of raises the question. I mean, how did we end up in that kind of bad situation that everything is kept secret and that people kind of pheck their data, even though they should know that it doesn't work that way? And so how did we get here?
30:21
And so I think what we can all imagine is it's not like people make that conscious decision, oh, I'm going to pre-register my study. Oh, no, no. Instead, I'm going to pheck it to hell, because I think we can assume that most people in science in general, but also in psychology, want to do proper science.
30:41
And so one possible explanation for that was offered in a paper by Paul Smoldino and Richard Macarith. And I think it kind of makes a lot of sense. It's the natural selection of bad science. So why would there be natural selection in science? Well, you need three preconditions to have natural selection to occur, and that is heritability,
31:00
variation of type and competition for limited resources. And so we do have heritability in some sense in science, because, for example, I learn how to run studies and how to analyse data from my advisor. So he passes on his knowledge about how science works. And then there certainly is variation of type. So some labs are already doing these like really rigorous studies where they like triple check everything and
31:24
run study after study to make sure they are not just like phecking without wanting to do that. And there are other labs who are probably less rigorous. And there certainly is competition for limited resources, such as faculty positions, but also grants and so on.
31:42
And so what now happens is we kind of start out with senior researchers and some of them will teach low-effort research just because that's what they are doing. And so that will lead to junior researchers who also do that like low-effort research. But now the crucial thing is that low-effort research, those not good sciences, will probably allow you to publish more.
32:04
You get more papers out there if you run very small studies instead of doing really rigorous studies. And now we have selection by publication of number, by number of publications. So those researchers who published a lot, oh, wow, this is like 100 papers, isn't this impressive, they
32:21
are more likely to end up with their own lab and will end up getting a lot of resources. So they actually end up being senior researchers who have a lot of junior researchers to whom they can like pass on their ways and their secret tricks. And so like this, you can imagine over time, if the system works like that, the
32:43
effort will go down and in the end, you end up with everybody doing low-effort research. And so actually what Richard told me is when they did this study, they kind of thought, now we tweak the whole thing and add replications and show that replications can save science. So the idea was you like change the simulation in a way that half of the studies that are being run are replications.
33:06
And if something you did does not replicate, then you get a penalty. And so they thought this could like change something about the way that effort develops. And so on the x-axis here, you just have like time. So this is just like iterations of that simulation and you have effort.
33:21
And actually, it doesn't really matter whether there's that like number of replication, like 50 percent of replication is actually just crazy, but even that does not prevent the effort going down. And so this looks kind of dark because it kind of predicts that no matter what we do, it will go down. And so what can we do?
33:41
And it's so one thing is, of course, think hard about how we select people, because the number of publications, you can imagine that it's probably not like the best thing to measure that. But it actually means like think really hard about it, because if you just change it and okay, we are not just looking at the number of publications, but probably also the impact factor, well, you can imagine that that again just goes wrong.
34:01
And so you probably have heard of Goodhart's law, so when a measure becomes a target, it ceases to be a good measure. And I think this is like a general truth. And I got an example from that super nice book, Argorithms to Live By, and it just seems like an instance of that human tendency to overfit. So take, for example, fencing. So the original idea behind fencing is that you are able to defend yourself in a duel.
34:24
So it's defencing. But then they turned it into a sports and now they have electronic scoring equipment, so little buttons at the top at the tip of the blade. And so you just have to kind of like poke your enemy to get a point. And so obviously. Fences will now try to optimise this number, so the result of the electronic scoring equipment, and though the scoring
34:47
equipment kind of correlates with what you were originally interested in, like defending yourself, it's not exactly the same thing. So what will happen is that people start to fence in a different way, so they will have like some quirky moves just to get more points.
35:02
And that kind of defeats the original purpose, though it's still an interesting sport. So that's probably not too problematic. But now if you apply that to science. So our scoring equipment might be the number of publications, also the impact factor of the journals that we publish in. And so it certainly does in some way correlate with scientific productivity or the quality of our science, but it's not exactly the same.
35:25
So what people will obviously do is try to maximise that scoring number. So they will try to publish as much as possible in the best journals. And so just like for fencing, this results in something that might look elegant and give the impression of fencing, but it does not serve the original purpose anymore.
35:41
It does not lead to good science anymore. And so this is like, I think, one thing we have to think about really hard, because I don't think there's an easy solution to that. And so another thing that is in a similar vein is grant culture. So if you think about it, so like our current grant culture rewards researchers who oversell their
36:04
research, because that increases your chances to get a grant and also kind of reward spending a lot of money, because if you want to get large grants, you need to do expensive research. And having a lot of big grants kind of looks really good in your CV. And if you look at that from a like a bird's eye view, then it's like really weird, like this is set up to go horribly wrong.
36:26
And so there's been a pretty nice paper by Scott Lilienfeld on the like just like complaining about the side effects of grant culture and psychology. And it's a lot of things that have a lot to do with the current crisis that is, for example, it heightens the incentives or QRPs for these questionable research practices.
36:43
It also disincentivizes direct replications, because you normally do not get money for that. And it kind of just like diminishes your time to think deeply, because it causes so much overhead, not only for the applicants, but also for the reviewers. And there's that huge administrative burden.
37:02
So the whole grant system really doesn't seem to work very well, and it kind of seems to like hurt science and at the same time be very expensive. And so there have been multiple proposals how to fix that. For example, there was this suggestion for an alternative funding system. So the idea is kind of so you have a certain budget and you actually just distribute it among all researchers.
37:24
So everybody gets the same, but everybody has to pass on half of it to somebody else that they are doing is good research. So scientists decide themselves who gets funded and they're doing that now as well. But now it's with those weird like review on boards and like this, it would become also way more transparent.
37:41
And the thing is, if you get a lot of money from the others because they are thinking you're doing fine research, that means you will have more money. But it also means that you still have to pass on half of what you get. So you also get to distribute more money. And so that is one idea to go from these like funding agencies to scientific agency.
38:00
And there have been other suggestions how to fix that. So, for example, just like fund everybody. Fund based on merit. And you can probably also already imagine what could go wrong with that if we are not quite able to measure merit. But also, like one idea that kind of makes sense to me is just filter out the worst proposals because some are really crappy. But you can't really say, oh, is this the top one percent or the top five percent?
38:24
We are not that precise in our judgments. So you could just submit the rest to a lottery. And one other thing that would probably already solve a lot of issues is starting to ignore grand portfolios and hiring, promotion and evaluations, because it's just like that really weird thing that you
38:43
need to prove that you are able to spend a lot of money to, for example, become a professor. And so it's kind of really sad. So I know some people who do like really, really, really good and rigorous research. And the only problem they have is that they are using existing data sets like the Biobank from UK.
39:02
And so it kind of means like, ah, it's not quite competitive on the job market. You're just doing too much good research for too little money. And so I think that would be one thing we could easily fix. So putting all these things together. So this is where we want to go. And there are many different ways we could try to get there.
39:23
So, for example, we could have single researchers who commit to change their practices. And we do have that in psychology, which is great. But we also need institutions to change, for example, like how do you phrase your job ads? And there have actually been now some job ads in Germany that explicitly called for applicants that can demonstrate that they did open replicable science.
39:46
So this is, I think, a huge step because it changes the incentives. And it's also, of course, departments who might have to rethink their hiring criteria. And it might also be journalists who have to think about which papers do we want? What do we accept? Do we really only want the flashy findings or maybe do we want to publish actual research?
40:06
And it's also, of course, about scientific associations because the way they are handing out prices right now, for example, for junior researchers might send the wrong message. And taking it even to like a higher level, of course, it would be nice to have funders on board.
40:23
And this is, for example, happening right now in the Netherlands, who are really progressive with these things. So the Netherlands Organization for Scientific Research now has like some money that is specifically for direct replication of previous findings. And so it's not much, but I think it sends that signal that we do value replications and so on.
40:45
And so, as you can imagine, I don't know whether this is like a general law, but there's already some disagreement between the people who want to improve psychological science. So we are already getting into like tough wars about how we do that best.
41:00
And I actually think that's a good thing that we have some disagreement, which are like the most important steps, because I think it just stresses that we cannot expect an easy answer. And that's probably a good thing, because that whole checklist mentality, oh, yeah, the p-value needs to be below that. And then it's a finding or like an applicant has to have 10 publications, then it's a good applicant and so on.
41:22
That whole checklist mentality got us into these troubles in the first place. And I think now it's up to us to realize that it's way more complex than that and that it's probably way harder to do proper science than we thought it is. So now I'm happy to answer your questions, if you have any.
41:43
Or tell me that psychology is not science.
42:43
OK, so the question was whether like fixing that issue about statistical power and increasing statistical power wouldn't that solve a lot of these issues? And I agree that it would solve quite a few issues, including that p-hacking, because it becomes unnecessary.
43:01
But it's kind of limited in that sense. So there are things like just like confounders and bad designs and so on. And even if you have like a stable confounder or a design flaw, like even sampling millions of people won't fix that. And so, for example, I'm working with fairly large samples, but that actually kind of makes the issue of statistical confounding even worse.
43:24
So it doesn't solve everything, but some parts certainly. There's a statement, I guess, that any metric that you use the moment it becomes a target becomes absolutely useless as a metric.
43:50
So the question is, for example, impact factor, it's not necessarily a terrible metric as such, but it's just that the moment people start gaining it, it becomes absolutely awful.
44:02
So do you think like some of the metrics that you propose like avoid this or do you just have to rotate metrics every once in a while? OK, so the question is, if every metric that we use becomes kind of corrupted, then are there any metrics that solve that issue? Or do we just have to like change the metrics randomly or whatever so people cannot predict it?
44:25
And I think that's a really, really hard question. So I think there is that general truth, like whenever the metric becomes explicit, it will be gamed in some way and you cannot like catch everyone who's gaming that. So I know it's not very satisfying, but I'm a proponent of looking at applicants like really thoroughly.
44:47
And so this is kind of wacky because I'm a personality psychologist. So we try all these standardized questionnaires and like develop criteria for diagnosis and so on. But in some sense, I do think we cannot have hard metrics for all these things. So there will always be like the human part and the human interview.
45:03
And I think this is super important. And so this is really weird because we really are like, oh, no, we need objective criteria. Anything else would be unfair. But the objective criteria are unfair as well. So we need some sort of mix. And maybe so that's also like probably also bad advice, but not make it too explicit what you're looking for, because in
45:24
that second, people will try to pretend that they have that skill and then it gets hard to figure it out again. I'm nervous on the first one of the mic.
45:40
Thank you for your talk. I thought it was fantastic. I really enjoyed it. You touched a little bit about the hereditary nature of having a supervisor that teaches you. Do you have any thoughts or comments about the role of education in terms of best practice and things like that? So I think education is super important because we do learn a lot of practices during our undergraduate studies and so on.
46:04
And so I do think this is like one huge thing. We have to make sure that people know how to use these methods and have proper method teaching. And that is kind of problematic right now because, of course, through that whole like kind of
46:22
bad system, we do have quite a few people who teach methods who don't fully understand them. So we actually do have textbooks and psychology like stats textbooks that get things objectively wrong. And so we kind of have to fix that. But it's kind of hard to bootstrap it because we have so many people who really don't fully understand stats.
46:41
But there are many people giving like really great lectures. And I know for a fact that some people are like, I'm not going to try to fix the old people, right? I mean, they are going to die one day anyway. So I'm focusing on teaching and making sure that the next cohort knows how to do better.
47:11
OK. Just let me remember what I wanted to ask. Yeah. So we discussed a bit the the entire issue with metrics and the fact that as soon
47:20
as it becomes known as such, it's something that can be gained and can happen virtually in any scenario. Your proposition with this crowd based funding was interesting because, I mean, the way you try to circumvent stuff like that, for instance, in governments is not having like a centrally planned metric, but doing everything in a more free market system.
47:43
Crowd based system. My question was, do you realize that a lot of the things which you might suggest, such as mandating or giving points or stickers for free data sharing, much as I agree with that, or even worse, requiring or highly incentivizing people to preregister their studies?
48:04
Wouldn't you agree that this kind of contributes to the bureaucratization of science and to the establishment of rigid metrics, which not only can be gained, but work against encouraging creativity, simply because when you have a good idea, it might not fit the present metric system?
48:22
So I very much agree with that. And I also see that this is something happening right now. So I'm working with pre-existing data and now people were like, oh, that means we cannot get the pre-registered badge and maybe we can come up with something similar for our type of research. And so I think that is like a real risk that we are now just imposing new standards that are like just absolute and you have to do that.
48:44
And so it's kind of like that metrics problem again. So it's hard to fix that. I always think that common sense would actually get us a long way. Like, for example, of course, that person never preregistered, but that is not a bad sign because that person is using pre-existing data sets and so on.
49:02
So it really depends on how we communicate these things. And I know that there is now like a strong push to make it, oh, it's so easy to do that and you should really do that and it looks really good if you do that. But we should again, like, just make sure that people don't start to confuse those badges with the actual science happening. So it's a bit like that thing, like confusing the finger pointing at the moon with the moon.
49:23
So we have to make sure that people are still able to evaluate the science without just looking at the badges and counting how many badges you got.
49:45
Yeah, to take up the very last sentence of your great talk, I guess a majority here is rather from the heart or nerd sciences. And there is a risk that we look down to something like psychology.
50:01
And I would say we have a quality problem everywhere in science. Let me give you a few examples. In condensed matter physics, I recently started a collaboration with a mathematician and he checked my equations.
50:24
And that surprised me because over the last 20 years, about a hundred of co-authors co-signed my manuscripts without ever checking what I had been doing. Example two, a colleague over 20 years published 150 papers and accidentally I found out
50:46
that he used almost each of his experiments in two or three publications in parallel. Example three, last year I had to review a pile of grant applications and in two out of ten applications, I
51:08
found that the author, the applicant cited some paper by a third party to support the importance of his own research.
51:23
And I looked up some of these references and found those papers did not at all support the importance of the applicant's research but only cited him in passing.
51:40
And so these people trust that the reviewer will never look in detail into the application. And this is a huge problem for all of us and in the long run in dangerous the funding and the freedom we enjoy.
52:00
I very much agree. So I was just talking about psychology because I know most about this but all the issues with the whole system, for example, why should any reviewer really take the time to look it up? Well, there are no incentives, right? They push people to do that. So I think there is the danger that other sciences develop the same problems and many already have them actually.
52:29
Do you think it would be beneficial to incentivize people to publish their failed hypotheses and if so, how?
52:40
Yes, I think it's super important that people publish that just because many of our, so if we run a meta-analysis, we actually have to have the null findings as well, right? So it's kind of like a trade-off. So people are also arguing, well, we can't just publish everything because that would cause a huge chaos. So I think we have to think about maybe new ways of publishing that don't cause such a huge
53:05
issue. I mean, you probably all know like you have thousands of PDFs on your computer with findings, whatever. And so I think we need a solution to be better able to integrate findings so that you can just publish like I tried this, I found nothing and make it searchable so that other people trying to do that can also find that.
53:21
And so I think that like requires us to rethink how we publish because the current way that psychologists at least publish is super outdated. So it takes like years until findings are out there, only the significant stuff gets published and then it's like hidden away in a PDF behind a paywall. So we really have to rethink that and we probably have to somehow incentivize people to share like the whole story.
53:46
Right. So you also mentioned that a lot of the time applications for professors will depend largely on how much grant money you've been able to acquire in the past. Do you think that this largely stems from big US universities or other universities really using that as a slogan saying,
54:06
we have all the money, we are good universities and that this is actually a public misconception that money equals good science? And how would you propose to change that public conception? Okay. So it's a great question. I actually think that the universities, not only in the
54:24
US but also in Germany and UK, so it's kind of like running really low on money. So they actually need to have people bringing in money, right? So you need that overhead from the project. So that's a huge issue and then there's that huge prestige associated with the grants that is a huge problem.
54:41
And so I actually don't know how to fix that without overhauling the whole way that academia works and changing the way we fund universities because that might be one source of that problem. I think it's a hard question. I cannot give you an easy answer but I think like one very easy step would just to no longer explicitly look at how many dollars that person brought in.
55:06
Though I heard that universities have such a strong interest in that that they probably won't let you make that. So it's really constrained in that sense.
55:24
Thanks for your talk. Do you think there's a problem of something called moral hazard? The physicists just said that there's people who win that are kind of bold. They cite things in their own way and they get through.
55:45
And every now and then there's a person who's concerned, a referee who's digging into it deeper. And the worst thing that can happen to this person is, okay, he has to submit his paper to another journal. But there's no, there's just this incentive to being bold and then in the end the system supports these people with a number of publications.
56:11
And if there would be some moral standards, I mean there's no number that might grab this but it's an unfair game in the end.
56:27
Yeah, so that's a very good point and that is kind of like in some sense a tragedy of the commons situation. So where you can have like free riders that just kind of like cheat a bit but they don't get caught and so they reap all the benefits and so on.
56:42
And so it's actually, it's also in that like natural selection of bad science paper. So what actually happens is why effort still goes down if you have those, like even if you have the 50% replications. The thing is some people will not get caught, right, and so they will still like get through with making those bold things you said.
57:03
So just like hope nobody catches them. And so I think this is like a problem that comes up in many different contexts, right, so just people like fraudsters and so on. And so I think we can't 100% fix that because if we try to 100% fix that we will have a lot of distrust and so on and we can't operate anymore.
57:24
So I personally do think that actually like morals and ethics play a role in that. And I would always hope that the people who pursue a career in science are actually interested in science and not in pursuing that career. And so I think one thing is so what Brian Nozick said at a conference a few years
57:42
ago is like if you want to become famous or rich, well you're at the wrong place here, right. And I think one solution to that is to make sure that there are job opportunities outside of academia. So I think in psychology at least it's a problem that many people do not see job opportunities outside of academia.
58:01
And so they stick in academia probably for the wrong reasons just because they think oh it's easier than finding a job on the open job market. And so I think that might be not that much of a problem in other fields because if you want to make a lot of money or like have a career and you're a physicist you can find a job somewhere else. And so I think that is like part of the solution.
58:23
But then I agree that you, so there will always be those people who get by and it's kind of unfair. And I think we cannot prevent that there will be some people gaming our system. That's not satisfying is it?
58:40
Thank you for the talk. One more question. Do you actually think that science wasn't broken somewhere, somewhere? So basically, or put it another way, can you say when did the science became broken or was it broken from the beginning? So I think that's very interesting. What's kind of interesting at least about psychology is that it is a very young science.
59:05
So we have a very brief history, maybe just 100 years, so one might think. So that's kind of the problem that in the beginning it was kind of more qualitative and very different norms and so on. So it kind of broke on the way to becoming a proper science I guess.
59:22
And so it's hard to detangle that because it would say okay we've become more and more scientific over time just because we're a young science. But then something goes horribly wrong in that process. And so there are different, so some people also think it's kind of like psychology is kind of premature. So we're not yet fully developed like the theoretical framework and everything.
59:42
But we are already trying to imitate physics by having those stats and numbers and models and so on. And so I think that is one explanation that kind of makes sense. And if I actually, so it's not like papers got worse over the course of the last year. So if I go back and read papers of the 70s, that's pretty horrible sometimes.
01:00:00
There were people raising all these issues already, like in the 70s, so we could have known that for a long time, but we just now realize that it's a really huge problem. I just want to make a tiny comment on becoming famous and rich doing science. We all know that lots of people try to become famous,
01:00:21
but I just want to give this small fact. Paul Thompson at University of South California makes between half a million and a million a year paid by the university, so you can also be bloody rich doing academia. So I don't know what to say about this, but small comment.
01:00:46
Thank you so much for the great keynote and answering all the questions. Thank you for the cool discussion.