Open research is on the rise, but a lot of research information still remains behind closed doors. The Center for Open Science, recipient of the Institutional Award, advocates for more transparency and open access, training scientists, publishers and funders, providing technology and policy recommendations. With this multi pronged approach, it has helped create an open science community which is becoming ever more self-sustaining.
By Helen Albert
In 2011, a group of scientists from the German pharmaceutical company Bayer tried to reproduce published data for 67 scientific studies in cancer biology, women’s health and cardiovascular disease—around 64 percent of the results were unrepeatable. A year later, Amgen, an American pharmaceutical company tried a similar project to reproduce cancer studies and could only successfully do so for 11 percent. This problem was very much on cancer biologist Tim Errington’s mind in 2013 when he applied for a job at a newly founded organization, the Center for Open Science (COS) in Charlottesville, Virginia. Increasingly aware of the reproducibility issues in science both from his own work and studies such as those completed by Bayer and Amgen, Errington was excited by the opportunity to help solve the problem. His enthusiasm and experience paid off and he was offered a job the Center the next day.
The open science movement has developed significantly since the early 2000s, calling for increased transparency and improvements in access to scientific data. It also calls for more studies without notable findings to be published, as leaving these studies unpublished further undermines scientific credibility and makes replicating results even more challenging. “There’s the understanding that research is open, but it doesn’t actually occur that way, or at least not as well as it should,” says Errington, who is now Director of Research at the Center. “Study data, the protocols, and the experiments that people tried that didn’t work: all that information doesn’t really make its way out of the lab.”
“There’s the understanding that research is open, but it doesn’t actually occur that way, or at least not as well as it should.” Tim Errington
The COS was co-founded in 2013 by Brian Nosek, now Executive Director, and Jeff Spies, who was previously Chief Technology Officer, to try and tackle this problem. Nosek, a Professor of Psychology at the University of Virginia, persuaded a group of colleagues to reproduce the findings of 100 psychological studies. This first reproducibility project really got the Center off the ground when it attracted a seed grant of $5.25 million from what was then the Laura and John Arnold Foundation, now Arnold Ventures. Notably, the results of this study supported previous claims and showed less than half of the original findings could be repeated. Just before Errington joined, another project to replicate the findings of 50 landmark cancer biology studies was funded, the full results of which will be available later this year. There is also an ongoing replication project in social sciences that has support from the US Government.
“We don’t shrug our shoulders at the problem, we provide very concrete policy recommendations to universities, publishers and funders.” David Mellor
The Center’s mission is split into five areas which include: training scientists, publishers and funders; helping to incentivize people working in these areas to adopt changes; building an open science community and building technology such as the Open Science Framework to help achieve this goal. While many people and organizations are involved in the open science movement, the Center’s unique multiple pathway approach to this issue makes it stand out, according to David Mellor, current Director of Policy at the Center. “We’re tackling a complex, system-wide process in a holistic, comprehensive way. We don’t sugar coat the extent of the problems”, he says. “But we also don’t shrug our shoulders at it, we provide very concrete policy recommendations to universities, to publishers, and to funders.” One such recommendation are the Transparency and Openness Promotion (TOP) Guidelines. Aimed at journals publishing research that has gone through review by other scientists, the TOP Guidelines include eight transparency standards. These deal with different parts of article publication such as registering studies before publication and making data analysis methods open for readers to review. Each of these standards is split into three levels, with level one showing moderate and level three excellent compliance with the transparency standards.
“The status quo for the vast majority of scientific journals is for articles to say, ‘you can call us up and ask for the data and we’d be happy to provide it’. It almost never happens, and even when it does happen, underlying evidence is almost never actually provided,” says Mellor. “Our recommendations are that there should be a specific disclosure, whether or not data are available. That’s what we call the level one approach. A better approach is that it just needs to be made available in a public repository. Our level three approach is based on not only making the data available but also on ensuring a third party checks your answers.”
The guidelines have been a success and the number of publications signed up to use them has risen from less than 600 in 2015 to over 5,000 in 2020. “I think it’s a positive way to showcase those journals that are taking great steps,” adds Mellor. Another positive development for the Center is the increasing use of the Open Science Framework (OSF), its online web platform built and maintained using open- source code. Cumulative user figures have jumped from less than 400 in its first year to almost 400,000 eight years on. The OSF is free for individual researchers to use and is designed to help them to carry out different open science actions such as data sharing, study preregistration, collaboration between researchers and a preprint publication server, a way of quickly publishing research before it undergoes any form of external peer review.
Lisa Cuevas Shaw recently joined the Center as Managing Director and Chief Operating Officer after a long career in scientific publishing. She explains that the purpose of the organization is not to police scientific research, but more to offer everyone involved a useful tool to enable more transparent practices. She adds that blaming researchers for not following open science principles during their research is rarely effective. “We really want to avoid that. It’s understandable why things aren’t enacted, due to either resourcing, lack of awareness, lack of any norms or understanding of what it could look like, lots of different things. And then again, the reward system is still what it is, which is publish a lot, publish your novel findings, not null results ... it’s a vicious cycle that researchers are a part of.” Errington says he has really noticed an improvement in knowledge about open science principles in the academic community over the last seven years. “It shows that we can have an impact. If you keep doing this and working with communities. The adoption occurs in front of our eyes, once they get convinced, they’re kind of excited. And then they come back to you.”
“We want to get stronger at how we can best maximize and leverage relationships in this system.” Lisa Cueva Shaw
The COS is now looking to the future. Nici Pfeiffer, Chief Product Officer at the Center, says that a future goal in her area is to help funders see if their open science policies have been complied with. “If we’re saying we really want open data, then how are we making sure that that is happening with studies they funded? We’re supporting that by adding some new features and workflows in our platform.” The management team at the Center has recently completed a plan for the next three years and identified some new areas it wants to focus on, according to Shaw. “We want to get stronger at how we can best maximize and leverage relationships in this system. And then the other area that we’re really focused on is how do we design things so that they’re self-sustaining by the communities themselves? That’s a big emphasis for us,” she added.
“It shows that we can have an impact. The adoption occurs in front of our eyes, once researchers get convinced, they're kind of excited. And then they come back to you.” Tim Errington
Bringing in more funding to expand their mission and increase visibility is also important. This September, the Center received $18 million in funding from Flu Lab, a private organization funding research projects and organizations targeting influenza. Following the upcoming publication of their cancer biology project, they plan to start work in this area.