Details

Intro: We have an incentive structure where people get rewarded for publishing in the top journals and publishing a lot. And we're not really rewarding people who do sort of rather more slow, cautious, careful research. And so I think the consequences are that we now have a scientific literature, which quite a high percentage of the work in it, I don't feel I can believe. AskDifferent, the podcast by the Einstein Foundation with Leon Stebe. 

Leon Stebe: What does quality mean in research? How do we improve the quality in science? And what is the best approach to foster research integrity? Dorothy Bishop stands up for open science and research reproducibility. She's a Professor of Developmental Neuropsychology at the University of Oxford. Her research is concerned with trying to understand the nature and causes of language impairments in children. And Dorothy Bishop is member of the jury of the Einstein Foundation Award for Promoting Quality in research. What a pleasure. Welcome to this episode of AskDifferent. 

Dorothy Bishop: Thank you very much for having me. I'm really looking forward to it.

Stebe: What does it mean for you, being a good scientist? What does that mean? 

Bishop: Oh, I think just being accurate, being careful, being self-critical, and caring more about the science than about yourself, I guess, at the end of the day. 

Stebe: So how self-critical do I have to be as a scientist? 

Bishop: Well, I think as a psychologist, I got very interested in the psychology behind this because I think being a scientist means suppressing a lot of natural human tendencies. And one of the natural human tendencies that we've got is something called confirmation bias, which is a tendency to only remember and seek out evidence that agrees with your prior viewpoint. And I've come to the view that to be a good scientist, you have to learn to sort of reverse that and really continually think if you find something that agrees with your views, you should think there's probably something wrong, there's probably another explanation. And always try and sort of look for the contradictions rather than just get excited and try and support your own views. So I think that doesn't come naturally at all. I mean, it's why we have such a lot of problems with science is that I think it needs to be taught. And in some disciplines, I think people really aren't taught it. They're just taught to a lot of facts. And then they go off and think, I've just got to gather more facts. But we really need to be taught a mindset that is going to move science forward by always being critical and, you know, being your own self-critic as well as criticizing each other. 

Stebe: What are the consequences for the research society if science is not that self-critical?

Bishop: Well, I think we have got in quite a mess. And I think what's happened is we've got a situation currently where not only, many people not really trained to be self-critical, but also we have an incentive structure where it actually encourages people to … they get rewarded for publishing in the top journals and publishing a lot, publishing things that are exciting and really sort of look very novel and groundbreaking. And we're not really rewarding people who do sort of rather more slow, cautious, careful research. And so I think the consequences are that we now have a scientific literature, which a quite a high percentage of the work in it, I think I can't I don't feel I can believe. And that's really quite serious. I think it's inevitable that people will take wrong paths and that, you know, that's how science progresses. You go down the wrong path and then you correct. But I think it's become harder to self-correct because we have this sort of mountain of literature, much of which is really not trustworthy. 

Stebe: So it's the pressure just to produce a headline in a scientific journal? 

Bishop: I think that's one factor. I think also, I mean, even without that, I think we would get things wrong. But also, we're not rewarding the right things. We're not we're not really so much thinking about rewarding people who do things carefully. And also, the other thing about the current ethos is in many places, it's highly competitive. And so, people really don't collaborate properly. They're continually competing to get ahead. And again, that can drive scientific progress. Sometimes, you know, they're very good examples of that, where people have progressed precisely because they've got a competitor. But it sometimes gets to the point where it's silly because we don't collaborate and each person only works on their own area and tries to continually edge ahead rather than in situations where it might be helpful to work in a more collaborative fashion. 

Stebe: Some already speak of waste in research, which sounds quite harsh. Waste. Would you use this term for research with no evidence as well? Is it waste? 

Bishop: Yeah. I mean, if you're doing studies that are designed in a way that is not going to be particularly reproducible and or which may be too small to give a conclusive result. You may have been better off waiting and doing a bigger and better study or a collaborative study. So, I think there is quite a lot of waste in research, which can come from people also not necessarily checking if something's already been done before, always sort of feeling that they've got to say this is all terribly novel. So, I think there's rather a range of different sources of waste, which really can mean that we're not using the resources we have got in the optimal fashion. 

Stebe: So how to fix it? What needs to be done to foster research and to foster research integrity?

Bishop: I think both bottom-up and top-down things can happen. I've mentioned already the incentive structure, and I think that that does need to change. And I think it is changing. And the people who can have the biggest influence are going to be the funders. And the interesting thing is that, certainly, in the UK, and I think in other places, people who fund research have got more and more aware of the problems. And from their perspective, it's terrible. They do not want to fund research that's wrong or that leads off in a misleading direction or that is just duplication of waste. So, they are getting more and more motivated to consider other criteria than just, is this terribly exciting? So that's one source. And then the other source is really training. And I think very much more the bottom-up influence, how we train early career researchers. They're the other group that are really highly motivated because they are, if you like, the victims otherwise of a system that doesn't reward the right people. And they get demoralized and leave if they really want to do good science and are continually being pressured to cut corners or to publish something prematurely or to embellish a result. So I think training is one thing. Training in ethics, but also sometimes just in methods, I think a lot of problems arise because people are using, for example, statistical techniques without really understanding them. So training on one side, but on the other side incentives.

Stebe: Now you are one member of the jury of the Einstein Foundation Award for Promoting Quality in Research. Why is this award so important? 

Bishop: Well, I was really pleased to be asked to be on the jury because I think this is like a superb way of showing that incentive structures are changing because we see always people getting awards for, you know, doing, you know, often very good research. But it's very much sort of the impression that you have to get some big flashy results. And that is the thing that needs to reward. Whereas the Einstein prize is saying what we should be rewarding are people who are boosting the credibility of research and making research quality a very high priority. This is, of course, not to say that other awards like Nobel Laureates don't do high quality research. But I think it's sending a strong message that there is an importance to research quality. And it's a message that's at all levels because the award has got these different levels of both a sort of more senior person and more junior awards as well. So, I think that is really … and it's attracted a lot of interest. We had far more people putting themselves forward or being put forward than we had expected. And so I think it's captured a good moment in science. 

Stebe: So this award should be an incentive for all scientists, especially for the young researchers, students, scholars? 

Bishop: Well, I think I hope so. I mean, I think it does show that we are valuing not just people who, after a very long career, have made really big steps forward. Although, of course, those people are really important. And they're relatively rare, you know, to find people who have been standing up for improving quality. But, yes, I do see the younger generation as very much the future. And whether or not we do overcome all the problems we've currently got is going to depend crucially, really, on people who are currently, I think, early career researchers, because sometimes they have to sort of be the ones that are pushing for methods to change.

Stebe: What would be your advice to young people who are aiming at a career in science and research regarding quality and research? 

Bishop: It's a very sad fact of life that very often if I talk about this, I do get early career people coming up to me and saying, well, I'd like to do what you're suggesting, but either my PI won't let me or if I do this, it will be career suicide. And I think, you know, that that is in itself really worrying. But I think I would just say you have to hang on to why have you gone into science in the first place. You've gone into science, presumably, because it fascinates you and you want to make a real difference. And I have seen numerous examples now of people who have really, you know, hung on to that and stood up and been prepared to sort of really ensure that methods are good, ensure that they do their science openly and reproducibly, and have benefited from it. I don't think it is career suicide. It may vary somewhat from one part to another. But I think my view is if it really were the case that you can only get a good career by cutting corners or being dishonest, you don't want to be in that career. So, it's important that we stand up for what we really believe in. And I think you then feel much better about what you're doing. It is almost also a mental health issue because you see people who are really conflicted if they feel they can't do science properly because of the pressures upon them. So I would always say do what you feel is right. And, you know, I think it won't be as bad as you imagine. In fact, it might be to your benefit. 

Stebe: And what is the difference between being a scientist nowadays and working in research decades ago? Is it challenging, more challenging now? 

Bishop: I think it is. I mean, I started decades ago, and it was very, very different. And, you know, the generation before me, certainly, you know, we have some top scientists who were in the UK who were like role models, but they worked in small groups. They often were very hands-on in the science that they did and would take ten years, twenty years working on a problem to really sort of sort it out. And certainly, we do have people in the UK who are Nobel Laureates who say these days they wouldn't survive. You know, they would be kicked out for being insufficiently productive, which is very worrying. So, I think I mean, there is a movement now in I think started in Germany, in fact, for slow science. And the idea that we've got to slow down a bit. We've got to take a step back instead of always rushing for the next important thing. And certainly, that was more the ethos when I started out. And in my time, I was regarded as reasonably productive. But again, by current standards, number of publications and so on was very small. But I did think hard about each publication and, you know, put a lot of work and effort into it. And I can't understand how these days people are churning out loads and loads of papers, you know, with so little time gone into each one. So, I think it has changed, but I think you can still survive if you try and press back and focus on the quality of the work rather than the quantity. 

Stebe: Do you think that the partnership between Oxford and Berlin is able to improve quality and research? 

Bishop: Absolutely. It's been wonderful. We've had this summer school, apart from anything else, which I've been involved in teaching on. Where, of course, in Berlin, you have the QUEST Center, which is doing great work on open reproducible science. And in Oxford, we have something called Reproducible Research Oxford, which is part of the UK Reproducibility Network. And I think linking up those two centers has been really fruitful. And what's lovely about it is I've just had an email sort of saying that all the materials, all the coursework, and everything is publicly available. So, it's not just being good for Oxford and Berlin. But Oxford and Berlin together are really producing things that can benefit many other people. So I think it's been a phenomenal achievement and a relatively low cost one, too. And a lot of the work has been done by really enthusiastic people who are not just sort of going through the motions, but people who are as committed as I feel about trying to really change things. It's been very, very refreshing. 

Stebe: Why are you so active in this field of open science and research integrity? Why do you stand up for this? 

Bishop: Well, I guess for the same reason I say other people should. I mean, I feel very firmly that science is a really important thing for the human race to get right. And when I've seen examples of people I mean, occasionally you see examples of outright fraud, which make me very, very cross. But more often, I just see examples of people who really sort of get things wrong and don't quite understand what they're doing and so on. And you feel things could be improved so easily with a little bit more attention to that aspect, with more understanding of what the problems are. So I get quite passionate about just it's almost, I guess, for me, like a religion. And I sometimes feel that this reproducibility movement is behaving a bit like a religious movement because people who get involved in it get really sort of quite enthusiastic and keyed up by meeting other people who see the same problems and want to fix them.

Stebe: Now we are still in the midst of a pandemic. Science has played a key role. Everybody was talking about the scientific progress in the last months. Do you think this public attention was good for quality in research? 

Bishop: Yes and no. I think there was an absolute tsunami of bad research, unfortunately. I was a little dismayed. As soon as COVID really started being taken seriously, a number of funders said, oh, we're gonna have special funding for COVID related research. And you saw a lot of people who knew nothing about COVID leaping in and trying to distort their research that it would somehow fit so they could get some funds. And there was a lot of work that was not good. On the other hand, you know, for people who already were in this area, epidemiologists or people developing vaccines and so on, I mean, it was just phenomenal to see how the skill and experience that those people already had come to play a huge part in sort of really getting things right. So, it was very much a two edged sword. But I felt myself that it would have been better if, in this case, funding had been distributed more to tackle, you know, if we'd first identified what are the big questions we want answers to and actually then assembled maybe multidisciplinary teams from many locations working together to solve those rather than giving lots of money to try and sort of tweak their research to fit in. I think that was a mistake and that more top-down research, you know, dictated whether question was dictated maybe by our research councils or even governments in this instance, pandemic situation would have been much, much more sensible. 

Stebe: And we have seen distrust as well or a lack of confidence. There was a suspicion in some parts of the population as well, when it comes to science and COVID or climate change, for example. So that's why quality and research is so important as well, to be accurate, to be trustworthy. 

Bishop: Yes. Absolutely. And sometimes people say to me, well, we shouldn't talk about the problems because, you know, we will destroy public trust. But I completely disagree with that. I think it's very important to show that we are aware of problems, that the people who, you know, are important in science are fixing the problems, identifying them as soon as they can. And science is, in that sense, self-correcting. Because I think the worst thing you can do is to try and cover up the problems that there are because that will just feed people's suspicions. But it is unfortunate that there is such a lot of, I think there's a lot of fake information actually being actively promoted by people with peculiar vested interest because it is just so bizarre, really, that I read a book recently by the women who developed the Oxford vaccine, very interesting accounts of how they developed it, called Vaxxers. And they said that, you know, they expected many, many problems on the way, many of which they did encounter. But the one thing they did not expect was having developed an effective vaccine to then be accused that, you know, they were injecting peculiar things into people and trying to kill people and all the sort of negative threats and things that they got. I mean, it is remarkable that that should have happened. 

Stebe: Now I would like to talk about your research work. Your recent work has been particularly focused on children, the specific language impairment. You have shown that there is a strong genetic component to these disorders. How do you gain your understanding of how genes affect language learning in children? 

Bishop: Yeah. That aspect of the work, it comes very much from twin studies because you find that these problems run in families. And the usual assumption that people make is that it must be that the parents, you know, don't talk to their children well enough and therefore the children don't develop language as well as they might. But with twin studies, you've got a natural experiment where you can dissociate the effects of genes and the effects of environment to some extent because you have twins who are either identical or non-identical growing up together in the same household. And it was just very striking when we started working with these children that you could see if you had one twin and they had an identical twin, that then almost always both of one if one had a problem, the other almost always did. If they were non-identical twins, you could see them like chalk and cheese. You could have one twin with a language problem. The other, entirely no problems at all. And yet, they were growing up in the same household. Now, that's just an informal account, but you can do much more formal modeling, obviously, more quantitative analysis of this. And show that there do seem to be some aspects of language development where there is a genetic basis. Having said that, it's not a single gene. It's not, you know, this is not a condition where you can just find one gene and therefore you've explained it. Increasingly, the evidence suggests that you've got a whole host of quite small effects of different genes that just nudge people in one direction or another and which interact with environmental factors as well. So it's not simply that you can say this is a genetic condition. It's rather that some of the predisposition is genetic. But it's important to know that because otherwise people keep blaming parents who probably have the same kind of problems themselves with language as their children do. And, you know, but it's not necessarily that it's this sort of direct causal link. And this means that it may be less effective to try and just change the environment with children. It might mean that you need rather to understand more about what it is about language learning that they find difficult and tailor interventions towards helping them learn language, perhaps more, you know, by presenting things more with more repetitions or more slowly or slightly differently. 

Stebe: And we are talking about children, which is a very sensitive, very special field of research. And I assume quality in research is particularly important in this field. Right? 

Bishop: Yes. I think and in any field where you're working with people who have some sort of disorder or some sort of medical condition, you have to be very aware that, you know, these are not just numbers. They are people and you need to, you know, respect the fact that they're involved in your research and do research that I think is of use and value to them as well. 

Stebe: What drives you to be a scientist? What is your main motivation? 

Bishop: I don't know. It's just, it very much always fascinated me. Right from when I was very, very young, I got interested in finding things out. I enjoyed science at school. I have a sort of somewhat nerdy streak where I like doing mathematical things and programming and things like that. So it feels very natural to me to do that sort of thing. And I'm just delighted that it's possible to have a career in it. Because I think if I didn't have a career in it, I'd probably still want to do it, but nobody would pay me to do. 

Stebe: And we all know every scientist, every researcher has to deal with setbacks as well. Maybe the scientific work is not receiving the favorable outcome. There is no evidence or the results are not confirming the assumption. So what to do? 

Bishop: Yeah. Well, I've certainly had a lot of experience of that. People tend to assume that because I've been successful, you know, everything's gone easily. But I've had many research proposals rejected, many, many more than I've had accepted over the years. And I've had many papers rejected, and I've had many, probably all of my favorite theories have been wrong, as it turned out. But I think you just sort of you have to say, well, you know, let the facts speak. And certainly, you have to have some resilience when grants and things are turned down. To some extent, you can console yourself that there's a large element of chance in this. And I have sat on grant boards and I and, you know, I know that I've seen good proposals that haven't got funded and so on. So there's that. But there's also you know, you do get aware that you can have this mindset that you get very fixated on something and you don’t see the problems, which is one reason why I think it is quite important to try and set up a scientific system so that you do have a lot of criticism and peer review and so on, even though it's horrible to be on the receiving end when it's negative. I think without it, we would never make really half as much progress. So I think I just sort of try to take a philosophical view. And I've been lucky in that I've had relatively long term funding from Wellcome Trust for most of my research career. So it hasn't been that know, it's not been as desperate that I might have to leave the field. But I have had a fallback position if I did have to leave the field. So I am trained as a clinical psychologist, so I could have worked as a clinician if necessary. So I think having a sort of you know, to bear in mind, you might need to have an alternative career and not find that, you know, an impossible idea is helpful when you're really at a low ebb and everything is going horribly wrong. 

Stebe: Maybe we have to talk more about setbacks.

Bishop: Yeah. I mean, I know that there was a while that when somebody was instead of saying talking about your h-index, which reflects how many citations you've got of your papers, somebody invented alternative thing that was like the opposite where all your you somehow had notification of all your rejected ideas and or a negative CV. That was it. Where you put down all your failures. And I think few people were brave enough to do it. But the interesting thing was quite, you know, I think it made other people aware just how often that can happen. And if you can't cope with that, then I guess it's difficult being a scientist because rejection is a bit like being an actor or something. You know, you want to go on the stage. Well, very few people have a, you know, an easy path to it, but you do it because you love doing it. And, eventually, if you're lucky, you know, it worked out.

Stebe: And beyond your outstanding research, you have written crime novels. Is that correct? 

Bishop: I have. But that is the … I mean, that's just an outlet for me. They're not serious novels. They're not properly published. They're self-published. But I found myself … it was weird, actually. It was almost like lockdown. There was one year when I'd been in Australia for the winter, our winter, there summer. And I came back from very sunny Australia, and Britain was covered in snow, and nothing was happening. No. Because in Britain, you know, we're hopeless with snow. So everything was shut, and I was stuck at home in this very cold house after the sunshine. And I just found that I couldn't sleep because of jet lag. And I thought, what should I do? I don't want to work. I don't know what to do. So I thought I'll try and write a novel. And I rather enjoyed it because I was capturing Australia, which I love and have visited several times. And so I was writing about this Australian scenario in a place I like, Fremantle. And I found it was such a liberation because it was so much the opposite of scientific writing. Because I could make stuff up. If something didn't work out, I could just kill off a character. You know, it was really nice. I used to do it on holiday for a few, I haven't done one now for a while, but I may possibly do another one when I retire, which is coming quite shortly. But it's just fun. I just like writing. But I particularly loved being able to write without having to worry about, you know, how many words I was using and getting things absolutely accurate. But using writing in this very different way.

Stebe: But maybe there are parallels between being a scientist and being, for example, a detective in a crime novel.

Bishop: Not sure. Probably. I mean, yeah, trying to work things out. Yeah. Well, Sherlock Holmes was, of course, the famous detective who had that very logical sort of approach to reasoning things through and working out what must be the case. I hadn't thought of it that way, but I think you're probably right. 

Stebe: Because in both cases, there is a need for precision, accuracy, and integrity. Right?

Bishop: Yes. Absolutely. Yes. And thinking, yeah, thinking creatively as well, I guess. You know, not always just sort of going down the same path but trying to think outside the box.

Stebe: Thinking outside the box. Thank you so much, Dorothy Bishop, Professor of Developmental Neuropsychology at the University of Oxford and member of the jury of the Einstein Foundation Award for Promoting Quality in Research. Thank you so much for your time and for your inspiring views. 

Bishop: Thank you so much for having me. It's been a pleasure.

Stebe: And thank you for listening. My name is Leon Stebeand this was this episode of AskDifferent. Please tune in next time. Until then, tschüss and bye bye.