Unreliable Biomedical Science, and a Solution?

The Economist has a long, detailed, and readable piece about the difficulties of inferring anything from the published findings of biomedical science.  There are all sorts of problems that fall short of scientific fraud, including the the biases caused by industry-funding of biomedical science, the biases of unblinded raters who see what they want to see, and the biases of journal editors towards only publishing “positive” findings.  (I am particularly enamored with this graphic, which shows the fundamental problem of inference.)  It is rare for researchers to even bother to attempt to replicate prior findings, but when replications are attempted, they often fail.  

The Economist piece can be read as something close to an outright assault on empiricism, at least as we now know it.  In practical terms, it is prudent for physicians, patients, and payors to be wary of the findings presented in even the top journals.

One of the beauties of our scientific system is that it is wildly decentralized.  Scientists (and their funders) can test any hypothesis that they find interesting, and they can use whatever methods they prefer.  Likewise, journal editors can publish whatever they want.  While such academic and market freedom is attractive, it results in quite a hodgepodge of science, with replication studies and publication of null results being afterthoughts.  The NIH and NSF have in the past functioned to set an agenda and demand rigor, but as their funding wanes, the chaos waxes.

The problems are scientific, but any solution will be institutional (and thus legal).  I have argued for a partial solution to industry bias in my short article, called “The Money Blind:  How to Stop Industry Influence in Biomedical Science Without Violating the First Amendment.”  Independent scientific testing could be conducted by a neutral intermediary, which would pool funds.  In a similar vein, there is also a new project of the Science Exchange, called “The Reproducibility Initiative.”  This program offers to be the independent scientific agency, which attempts to validate known results.   But there is not yet a large-scale funding model in place.  If biomedical journal editors would at least put disclosures in their structured abstracts (an intervention we have tested), over the long run that may also nudge industry to use such gold-standard independent testing, when they have something that is truly provable.  And, at least in the domain of the products regulated by the FDA, the agency should consider using its current statutory authority to push companies towards independent, robust, and replicated science.

Be Sociable, Share!
This entry was posted in Christopher Robertson, Empirical, FDA, First Amendment, Pharmaceuticals, Research Funding, Research Misconduct, Scientific Evidence by crobertson. Bookmark the permalink.

About crobertson

Christopher Robertson is a professor at the James E. Rogers College of Law, University of Arizona, and affiliated faculty with the Petrie Flom Center for Health Care Policy, Bioethics and Biotechnology at Harvard. Robertson also leads the Regulatory Science program, a partnership with the Arizona Health Sciences Center and the Critical Path Institute. Professor Robertson's research focuses on how the law can improve decisions by individuals and institutions -- attending to informational limits, conflicting interests, and cognitive biases, especially in the domain of healthcare. Blending legal, philosophical, and empirical methods, Robertson's work has been published in the New England Journal of Medicine, New York University Law Review, Cornell Law Review, Emory Law Journal, and the Journal of Empirical Legal Studies. He has received research support from the Robert Wood Johnson Foundation, and runs the Law and Behavior Research Lab at the University of Arizona. Robertson graduated magna cum laude from Harvard Law School, where he also served as a Petrie Flom fellow and lecturer. He earned a doctorate in Philosophy at Washington University in St. Louis, where he also taught bioethics. For 2013-2014, he was a visiting professor at Harvard Law School, and will visit at NYU School of Law in 2016-2017. Robertson's legal practice has focused on complex litigation involving medical and scientific disputes.