Ever since we were all kids, starting about the fourth grade, we had to relearn a simple process every time we cracked our science book open for the first time that year: Observation, Hypothesis, Experiment, Collect Data, Falsify or Sustain Hypothesis, Report Results. The good ole’ scientific method.
Literally the bedrock of all advanced scientific research, the scientific method guides inquiry in fields as diverse as psychology, AI research, geology, virology and pharmaconeuropathology.
There are other important pillars of scientific research, one of which is replication. Replication is a peer-review process by which members of the scientific community will attempt to replicate the conditions of a colleague’s experiment and run it for themselves, thus making sure that the reported findings of a given experiment are objectively valid. The scientific method is used to confirm results, but only through rigorous replication trials can we be sure that researchers used the method correctly in the first place.
It is this important process which has been the center of controversy, as of the past two decades, as many sciences experience a so-called “replication crisis.” Metascientific analyses of studies in psychology, the social sciences, medicine, economics and other fields have revealed a startling fact. Many, if not a majority, of new research cannot pass muster when replicated, meaning that a startling proportion of papers published in scientific journals may represent scientific dead ends.
The problem is most pronounced in fields like psychology, which have been beset by studies that routinely fail replication, or have weak statistical results, or just downright contain questionable research practices.
But the problem isn’t just confined to psychology. In fields like artificial intelligence, researchers often can’t even attempt to replicate the results of studies by major companies like Google because major companies often refuse to make their code or datasets generally available, and most universities and small companies that would be doing the replicating don’t have the processing power of corporate-owned computers like Google’s Sycamore machine.
The replication crisis has immense impact in fields like the social sciences, where everything from psychological treatments to educational approaches could be built on faulty foundations, and in medicine where drug trials might yield false-positive results, thus costing consumers, taxpayers, and drug manufacturers potentially millions for a drug that does not work. Lack of reproducibility in new fields like AI is even more worrying because of the drive to put complex decision-making, like driving a car or running predictive policing algorithms, into the hands of machines that we don’t fully understand.
Fundamentally, the crisis in reproducibility goes back to three fundamental problems. Firstly, scientists don’t want to report experiments with null results, because they are less likely to be published in journals. In some cases, less than 10% of experimental results reported in journals were negative, meaning over 90% of the studies reported some significant finding. Second, scientists want to make new discoveries rather than reproduce old ones, which drives a lack of reproducibility studies in the first place. Lastly, it is far too easy to design studies where small sample size or certain statistical analysis tools can be used to provide false positive results.
Scientists in the hardest-affected fields are already starting to work on solutions to the replication crisis, reforming their disciplines from within. However, new researchers brought in as graduate students have an opportunity, post-crisis, to change the way that science is done for the better.
At its heart, the replication crisis is, unfortunately, about scientists letting other concerns like grant-funding, political or ideological biases, professional advancement, or an old-fashioned desire for fame get in the way of doing proper research. If the sciences are to help the public, the public needs to be able to trust their experiments. This is especially true for fields like climatology, where crises of reproducibility give ammunition to political groups skeptical of anthropogenic climate change. On such hotly contested topics like climate change, the public needs to have confidence that scientists are telling the truth.
Up and coming research assistants who will one day be scientists in charge of university research efforts or laboratories should seek to refine the honor of the discipline and abide by the principles of scientific inquiry, while expecting their colleagues to do the same. Efforts are already underway in every major field to remedy the causes of extreme reproducibility failure, but we can do better. For the scientific community to truly overcome the reproducibility crisis, a cultural change will need to take place across the disciplines, one favoring slow and careful procedural work, dedication to truth over publishable results, and a willingness to stand up to incentive structures from research companies and universities that conflate lots of research with good research.
Image Credit / LaboratoryNews.co.uk
