Science plays a pivotal role in informing the work of the social sector—not to mention business, government, and society at large. (Fun fact: A quick scan of our own grants suggests that since 2001 the Hewlett Foundation has supported almost 1,000 research projects with over $390M in funding—and these are just the grants with “research” in the title!)
Scientific facts are solid, trustworthy, reproducible. That idea, that the same experiments should always get the same results, regardless of who performs them, is in fact a cornerstone of science’s claim to “objective truth.” But as Mark Twain so eloquently put it “it ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so. ”
A growing chorus of voices is alleging that Twain’s jibe can be applied to a great many scientific findings, both in the natural (biological and physical) and social sciences. The journal Nature has developed an archive of reports and analyses called Challenges in Irreproducible Research. Articles from The Atlantic, The Economist, and The New York Times offer a few recent examples of the purported problem in the popular dialogue. There’s even a forthcoming Sony/Participant Media film, Merchants of Doubt, related to the topic.
Glenn Begley, chief scientific officer at TetraLogic Pharmaceuticals, described an experience he had while at the drug company Amgen: He and his colleagues were unable to replicate 47 out of 53 “landmark” papers about cancer—all of which had appeared in top journals. Evidently some of the results could not be reproduced “even with the help of the original scientists working in their own labs.”
While much has been made of the potential problems with scientific integrity, we need to hesitate before jumping to any firm conclusions. If studies cannot be replicated, it could be for two very different reasons:
Dishonesty is, of course, one possibility. Certainly there are incentives for researchers to discover “surprising” results. Academics need to get their work published in well-regarded journals in order to secure funding or tenure. Top journals’ rejection rates can exceed 90 percent and, unsurprisingly, studies with surprising findings are more likely to make the cut. Alternately, according to the aforementioned article in The Atlantic, “maybe sometimes it’s the questions that are biased, not the answers”—as with medical research studies comparing a company’s new drugs “against those already known to be inferior to others on the market.”
Second, and perhaps less obvious, even when researchers act with complete integrity, conventional practices common in their fields may produce findings that are misleading or invalid. According to The Atlantic, these challenges could range “from what questions researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analyzed the data, to how they presented their results, to how particular studies came to be published in medical journals.”
These kinds of concerns have inspired a number of responses, including:
- President Obama’s Council of Advisors on Science and Technology, which has begun to explore problems of irreproducibility in science;
- The National Academy of Sciences, which has convened an expert panel to address the issue;
- The National Science Foundations’ Social, Behavioral, and Economic Sciences directorate, which has formed a task force on the topic.
At the Hewlett Foundation, we’re supporting scientists at Rutgers and Stanford Universities, in collaboration with Stanford’s Center for Advanced Study in the Behavioral Sciences, to assess the prevalence of these problematic practices, alongside the feasibility and potential impact of proposed interventions to improve “scientific integrity.”
In the meantime, many scientific journals have recently adopted policies designed to elevate the validity and integrity of the research they publish. According to Stanford and Rutgers, these policies include requiring minimal sample sizes, providing additional space for methods sections in order to increase transparency, inviting authors to complete checklists of high integrity practices (e.g., reporting all variables, all conditions, etc.), requiring authors to post their data online for others to be able to access and review, stamping articles with badges for high integrity practices (transparency, pre-registering hypotheses and methods to distinguish theory-testing from exploratory analyses, making data available online), and setting aside space to publish attempted replications.
It remains unclear how big the problem really is. We hope the research we are supporting will help to … um… clarify whether the research our researchers are researching is flawed. In the meantime I’m looking forward to Merchants of Doubt — a comedy, I think?
