Most people have set pragmatism as their
default position on bioethics. If it works, why not use it? If human embryonic
stem cells are reported to be effective, for instance, what harm can there possibly
be in using them? In fact, it may be immoral not to use them after the incredible
progress reported in this week’s issue of Nature (or Time or the National
Inquirer)!

But in an era of science by press release, pragmatists
should know how reliable such reports are. And respected studies into the
credibility of all medical research – not just on stem cells – suggest that
claims of incredible advances are precisely that: incredible. In fact,
according to a leading medical statistician, Greek academic John Ioannides,
“most claimed research findings are false”.

Dr Ioannides is not a crank or an enemy of
science. On the contrary, his work has been published in leading journals and
his claims are widely accepted among his colleagues. He has worked at Harvard
University, Tufts University and Johns Hopkins University. His ground-breaking
2005 paper in the journal PLoS Medicine has become the most downloaded in its
history. Every year he receives hundreds of invitations to speak at
conferences. “You can question some of the details of John’s calculations, but
it’s hard to argue that the essential ideas aren’t absolutely correct,” Doug
Altman, the director of Oxford’s Centre for Statistics in Medicine, told Atlantic
Monthly.

Ioannides’s claims are largely statistical
and thus require much brain cudgelling for laymen. But his conclusions ought to
rattle anyone: that “most research findings are false for most research designs
and for most fields” and “claimed research findings may often be simply
accurate measures of the prevailing bias”.

Why is this?

There are a number of interlocking reasons.
Many studies are too small to be reliable. The best ones involve several
thousand subjects, but many studies, especially in genetics, are based on fewer
than a hundred. Many studies are badly designed or are hard to compare to other
studies of similar data.

Prejudice plays a role as well. It’s not
necessarily ideological or financial; old-fashioned chest-beating,
turf-protecting arrogance is just as effective. Scientists who are committed to
a theory are less likely to find contradictory evidence. “Many otherwise
seemingly independent, university-based studies may be conducted for no other
reason than to give physicians and researchers qualifications for promotion or
tenure… Prestigious investigators may suppress via the peer review process the
appearance and dissemination of findings that refute their findings, thus
condemning their field to perpetuate false dogma,” wrote Ioannides in his 2005
PLoS article.

And finally, “The hotter a scientific field
(with more scientific teams involved), the less likely the research findings
are to be true.” Ioannides attributes this counter-intuitive effect to
cutthroat competition among scientists to publish exciting research first. “This
may explain why we occasionally see major excitement followed rapidly by severe
disappointments in fields that draw wide attention,” he says. Isn’t this
relevant to far-reaching claims made for embryonic stem cells?

Even more discouraging for medical
researchers is that the gold-standard of medical research, double-blind
randomised trials, are not altogether reliable either. In another
2005 paper published in the Journal of the American Medical Association,
Ioannides examined 49 of the top science papers of the previous 13 years. They
had appeared in the best journals and had been cited extensively. Yet between
one-third and one-half of them were unreliable because they were later found to
be either outright wrong or exaggerated.

None of this means that all scientists do
shoddy work or that science itself is fatally flawed. Science is a slow slog
towards the truth whose milestones are false intuitions and failed experiments.

But it does mean that politicians and
voters ought to be wary of early findings until they are repeatedly confirmed
by other researchers. Unfortunately, this is a process that may take years to
work itself out – far too slow for journalists who are searching for sound-bites.
But unless they convey the ever-tentative nature of progress in science, they
are deceiving their readers. As science journalist Joann
Rodgers, of Johns Hopkins University, says: “Part
of the responsibility for publicly communicating science is to help the public
understand that scientific truth is a journey.”

Although Ioannides’s analysis is widely
accepted, some researchers fear that they might be misinterpreted and used to debunk
science or to promote shonky alternative therapies. But he responds that the
truth is the best medicine: “The scientific enterprise is probably the most
fantastic achievement in human history, but that doesn’t mean we have a right
to overstate what we’re accomplishing.” Sound advice for every pragmatist!

Michael
Cook is editor of MercatorNet.