Keep Calm and Improve the System

The quality of science is currently an important issue. But has science ever been better than it is today? 

DB Yes, I think it has. In the second half of the last century, maybe even in the first half, we had fewer people carrying out a more careful type of science.

DI I only partly agree. Yes, science has become a victim of its own success. There was a time when there was little money, few scientists, and to become a scientist was not really the best career choice. Then — especially in the second half of the twentieth century — science became a huge business; a mass movement. But even though research may have been more effective then, it is now much more influential and important. One hundred or even fifty years ago, it would not have been possible to develop a vaccine within just one year. 

What made the quality of science, at least in some specialist fields, decline in particular?

DB To some extent, it is the incentive structure which has created a very competitive situation. Junior researchers are usually funded only for a short period and have to make their mark very quickly. They are often encouraged to cut corners because they do not have enough time for careful work. In addition, the publication system is really broken. It no longer plays a communicative role. Researchers do not publish because they have found something new and good; they publish for the sake of publishing—for their resumes. I hear young people in my discipline of psychology talking about publishing ten papers per year during their PhD studies. That is ridiculous—one per year might be realistic. And if you want to evaluate a scientist, you do not have the time to read what this scientist has written, so you have to rely on metrics such as the impact factor of the journal you publish in, which is not very useful.

DI It is the same with project proposals. There are too many of them, which are written too fast, and often the reviewers don’t have the time to read and evaluate them properly. It is difficult to decelerate the system, but we should do it.

Professor Bishop, you mentioned that young researchers are forced to cut corners. What do you mean by that?

DB If you think back to the big discoveries, they were made by people working intensively for several years to solve a problem. Now you have to get results quickly: sometimes that means that you publish something before it is really certain that everything has been done right. Sometimes it also means that you make things look exciting when they are not particularly exciting. Or you use problematic methods to squeeze something out of your data. Or you just talk about the part that allows you to tell a good story and then you do not mention other things. Some people do not have the time to check if the cell lines they are working with are contaminated. Looking through the literature, I find a lot of papers I do not believe or that I think are overhyped.

DI I do not think the situation is this bad. The intensity and speed of progress in certain fields today is much higher than ever before. But if you look at the output in relation to money spent, we can now afford to have a system in which high-quality output is hidden among a huge amount of waste output. To use an expression from biology: the system became eutrophic—like a lake overfed with nutrients. In a eutrophic lake, the actual biomass in the water increases, but biological diversity diminishes, and water quality goes down.

Is there an awareness in science that things cannot go on like this? Or is this a minority viewpoint?

DB It is a minority viewpoint. Because those who have benefited from this system are those who tend to be in a position of power, and so they are not particularly keen to change it. But the younger generation really wants to change things.

DI Again, I am less pessimistic. I have many colleagues who share the view that we should change things. In the past twenty years, there have been many initiatives developed by individuals and organisations that lead in this direction. The Einstein Foundation Award is one striking example. So, there is hope. What worries me is that some of the best PhD students, often women, decided not to stay in research because they are not willing to compromise in order to be successful. When I have discussed this face to face with PhD students who have decided against a research career, I always felt that I was caught in a dilemma: Should I convince them to stay because they are excellent or should I admit that they actually have a point? 

DB Yes—those are the ones we should be keeping because they are conscientious and careful scientists; the ones that carefully assess the situation and say they cannot do the right kind of science in this environment. We need to work on structures that prevent these people from giving up and leaving. 

It seems that during the pandemic many people wondered about the amount of discussion among experts. Did it damage the reputation of science that its progress could be followed in public or is this kind of transparency helpful?

DI As a scientist, I would say the system functioned quite well, with a few exceptions. But often the public had the expectation that science has to have an answer to every question and that scientists never change their minds. Maybe we did not explain how science functions well enough and that different views are common, especially in a new field. And that later on, knowledge becomes more and more consolidated. And maybe we have forgotten to point out that to say “I don’t know” or “I have changed my mind because I have new facts” is a virtue. 

DB There is a saying that you should not wash your dirty linen in public, but I think if there are problems, we should talk about them. What has harmed science more has been the tsunami of rubbish results that have been generated far too quickly and to some extent also fraudulently by people trying to make money. People who are skeptical about science now have the means to promote their opinion very strongly. Even as a small minority, they can produce a lot of noise.

DI The largest issue perhaps is that the media always wants to balance opinions. You have to have someone from a different point of view, even if 99.9% of scientists agree on a certain point. The public is then left with the impression that there is a lot of fundamental disagreement, even though this is completely disproportionate. That has done a lot of damage. 

Are certain disciplines affected more than others by the quality issue?

DB The hardest problems I am aware of exist in disciplines where you deal with probabilistic phenomena to explain complex findings. A lot of people are using complex statistical methods without any real understanding of what they are doing. In psychology, researchers were among the first to become aware of how bad the problem is. I think it is as least as bad in biomedical science. The only subfield that has done well is clinical trials. Here the problems were found quite early in the 1970s and methods were developed to deal with them. But if you look at preclinical studies, animal studies, biology, and ecology, the situation is still problematic, in my opinion, because it is not even broadly accepted that there is a problem. Issues around quality of research affect the humanities, too, but they are different in kind, and it is important to identify solutions that are relevant for each field. 

DI In physics and chemistry, there are difficulties as well, but the self-correction system usually works more reliably and faster. For instance, when 35 years ago, scientists in Salt Lake City claimed to have found so-called cold fusion, but in reality, misinterpreted some of their experimental results, within no time at all many groups started to do the same kind of experiments. Cold fusion was off the table in less than two years. In some of the “softer” disciplines, like the social sciences, psychology, and others, it is not so easy because it can take a lot of time to do the experiments and to reproduce them.

What is the role of the funding agencies in improving the quality of science?

DB They are the ones who really want to solve the issue and are heading in this direction because they do not want to waste their money. They have started to develop new criteria. For instance, research must be more open so that people can look at the results and the methods. When I give talks about the need for more reproducibility in science to young researchers, many of them say they would love to work that way, but they fear that their career will suffer. I try to show that it is the other way round, that their work will be respected when it is reliable. There are more and more individuals and institutions who fiercely defend reliable research, and who in turn are looking to employ and support those who share their committed approach. 

Is there a need for more meta research on science?

DI My impression is that meta science became too much of a philosophical task and did not do much to keep research clean. For me, the highest priority should be to keep the right people in research. The quality of research will stay high if we manage to attract and keep the right people in science.

DB I think we need more down-to-earth metascience, more documentation, and evaluation. Personally, I try to be a role model to show that yes, you can preregister your study to make sure its results are going to be published, regardless of the outcome. You can post your data openly; you can deposit your code so that others can have a look at it; you can actually work in this way. Of course, I do not have to worry about career suicide; I can just do it. But I see a lot of junior people becoming very enthusiastic about these ideas — they say it feels like doing real science. This way of working seems to be more satisfying for the researchers themselves, too.

As chair of the Advisory Board of the UK Reproducibility Network, you are engaged in a radical form of Open Science. Could you describe your perspective? 

DB Open data is important. But even if you have the data, you cannot necessarily reconstruct the results. You need to have access to the script, the processing steps too—not only to get your own research clear but also for the next generation that wants to learn how to do good science. I like the idea of open protocols, even using videos, showing, for example, how exactly a researcher shakes their test tubes. Sometimes there are really minor differences in methods people use and the descriptions are often not detailed enough. The more we can make clear and transparent what we did and what we concluded, the better it is. Of course, when I make my data or my code available, others might find an error in them. But instead of being afraid, I should be happy that the error has been found; there are always errors and that is not a source of shame. And if someone finds new things in my data, I very much encourage that. 

Will the Einstein Foundation Award help in the promotion of quality in research?

DB The Award is such a great thing because it sends out a big signal: Look, here is a funding institution with an international committee of experts and scientists demonstrating that what they really value is researchers who work in this particular way. By honoring those who are pushing for improvements in the nature of science itself, they are sending a signal to the scientific community across the whole world. I was astonished by the quality and quantity of the applications. It also helps us to build a community of people doing this sort of good science. And thanks to its international outreach, you get ideas from people working in different scientific contexts. I think the prize has a real potential to accelerate improvements worldwide.

DI It will definitively help and it is a fantastic chance for the Einstein Foundation and also for Berlin. Berlin is an important place for science, especially since all the universities are working together. And it fits really well: I think Albert Einstein is the perfect example of a scientist whose publication list isn’t actually really long but undoubtedly outstanding. He always felt strongly about the responsibility of research. Einstein would probably be unhappy about certain developments in modern research, but he would certainly be happy to see that his name is being used for a prize that advocates a fundamental change in doing research and that is boosting its overall quality.