Ben Goldacre

Science Kit for Ben Goldacre

Missing data.

Trials are the best tool we have in medicine to find out if a treatment works, or doesn’t. They are simple, fair tests: take a group of people with a condition; divide them into two groups; give one treatment to one group, and another to the other; then measure how many people get better in each group. Medicine relies on this evidence.

Unfortunately, when people conduct trials, they often leave the results unpublished, if they don’t like them. This means that we only see half of the evidence, which can be completely misleading. If I tossed a coin 100 times, but only told you the outcome from half the tosses, I could make it seem as if I had a two-headed coin: you would be a fool to let me get away with that. And yet this is exactly what we tolerate in medicine.

Fluke positive finding gets published. Refutations don’t.

A psychologist published research showing evidence of precognition in the Journal of Personality and Social Psychology. It’s possible that this was a fluke finding. Others tried to replicate the same study, and found no evidence of precognition. The journal declined to publish their study, saying they don’t publish replications as a matter of principle. This is a serious problem for research dissemination, as it means freak positive findings are more widely disseminated.

http://www.badscience.net/2011/04/i-foresee-that-nobody-will-do-anything-about-this-problem/

Molecular biology flukes get published.

This is not limited to precognition research! A group of researchers report in the March issue of Nature how they tried to replicate 53 early, laboratory, preclinical studies of potential targets for cancer treatments. 47 out of 53 could not be replicated. They recommend that there should be more opportunity for researchers to publish negative data, and shifts in the incentives for researchers to do so.

Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012 Mar 28;483(7391):531–3.

http://www.omsj.org/reports/Begley%202012.pdf

Anti-arrhythmics

In the 1980s over 100,000 people died through the practice of prescribing anti-arrhytmic drugs to patients who’d had a heart attack. We could have had earlier warning of the dangers of this, if we had seen the results of an earlier trial on a similar anti-arrhythmic drug that killed patients. Because the drug was abandoned, and it never came to open market, the findings of this trial were never published. The researchers published their results a decade later, explaining that they were now concerned their findings could have helped to avert disaster. This story is well told by the researchers themselves.

“When we carried out our study in 1980, we thought that the increased death rate that occurred in the lorcainide group was an effect of chance.”

“The development of lorcainide was abandoned for commercial reasons, and this study was therefore never published; it is now a good example of ‘publication bias’. The results described here

might have provided an early warning of trouble ahead.”

Cowley AJ, Skene A, Stainer K, Hampton JR. The effect of lorcainide on arrhythmias and survival in patients with acute myocardial infarction: an example of publication bias. International journal of cardiology. 1993;40(2):161–6.

This story is also well covered in the excellent book “Testing Treatments”, available free for download online (declaration: I wrote the foreword!).

And it’s not the distant past….

Reboxetine 
7 trials were conducted comparing reboxetine against placebo. Six trials showed the drug was no better than a dummy sugar pill. One trial showed it was better than a dummy pill.  Only that positive trial was published. Similarly, three trials showed reboxetine was as good as any other antidepressant, but data from three times as many patients showed it was worse. The unflattering data wasn’t published.

Eyding D, Lelgemann M, Grouven U, Harter M, Kromp M, Kaiser T, et al. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ. 2010 Oct 12;341:c4737–c4737.

Antidepressants

That’s an extreme example. How commonly does this happen? Very. If you want to document how often trials go missing, one good option is to find a list of trials you know have been conducted and completed, and then go to see if those have been published. For example, you can often get information on trials conducted before a drug was licensed, from medicines regulators. This isn’t all the trial data on a drug, but it’s more than enough to explore the phenomenon of unflattering data that goes missing in action before being published in the academic journals that doctors read. These researchers went and found all the trials on all antidepressants.

37 of the positive trials - all but one - were published in full. The trials with negative results had a different fate. Only three were published. 22 were never published. The remaining eleven trials - which had negative results, in the FDA summaries - did appear in the academic literature, but when they did, they were written up as if the drug was a success. In reality, we have 38 positive trials, and 37 negative ones; in the academic literature, we have 48 positive trials, and 3 negative ones.

Erick H. Turner et al., “Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy,” N Engl J Med 358, no. 3 (January 17, 2008): 252-260.

This is endemic.

A systematic review published in 2010 summarises all the studies ever conducted looking at the problem of trial results that go missing in action. They found that overall, positive results are about twice as likely to be published as negative results. This is an excellent and thorough overview of the literature on how often trials go missing, and why. It’s not a pop science read but it’s thorough!

Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010 Feb;14(8):iii, ix-xi, 1–193.

Tamiflu

The company refused to hand over the results in a form that the Cochrane reviewers were able to use. They have since got access to some of the full Clinical Study Reports, which has exposed possible discrepancies. This is a story to watch closely.

Doshi P. Neuraminidase inhibitors--the story behind the Cochrane review. BMJ. 2009;339:b5164-b5164.

The most current discussion on this story appears in the April 2012 PloS, in which they discuss the company’s explanation of why they won’t share the data, and whether they find these reasons credible:

Doshi P, Jefferson T, Del Mar C (2012) The Imperative to Share Clinical
Study Reports: Recommendations from the Tamiflu Experience. PLoS Med
9(4): e1001201. doi:10.1371/journal.pmed.1001201 http://bit.ly/HIbwqO

Fake Fixes

People have made various grand claims to have fixed this problem: this has been harmful, since it has left the wider academic and medical community with the false impression that there is no longer a problem.

This paper is a good account of the history of how people came to realise that there was a problem to fix, and how the fixes failed to materialise.

Chalmers, I., 2006. From optimism to disillusion about commitment to transparency in the medico-industrial complex. JRSM 99, 337 –341.

The ICMJE, the international medical journal editors association, said that they would force people to register all trials, by saying they would only publish pre-registered trials. Sadly, they didn’t hold the line, and continued to publish unregistered trials.

Mathieu, S., Boutron, I., Moher, D., Altman, D.G., Ravaud, P., 2009. Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials. JAMA 302, 977–984.

Since 2008, you’re supposed to post results to clinicaltrials.gov within a year of completion of your trial. It seems this rule has not been followed.

Prayle, A.P., Hurley, M.N., Smyth, A.R., 2012. Compliance with mandatory reporting of clinical trial results onClinicalTrials.gov: cross sectional study. BMJ 344, d7373.

Even if this fix had been implemented, it would not be enough, because it only requires publication of trials starting from now. We don’t practice medicine using only the results of trials starting from now! We need the results of all trials, for all drugs in current use, without exception, to reduce the risk of us inflicting avoidable harm on patients.

This isn’t about stifling innovation, or “attacking industry”: it’s about ensuring that we innovate, with our eyes wide open. Doctors and patients should be allowed to see all the evidence on what works.

Further reading:

This is an excellent academic overview of the need for all trials to be published, from an open access journal.

Gotzche paper (free online)

http://www.trialsjournal.com/content/12/1/249/

This is an excellent book on evidence based medicine, and the problems in it. It is written by eminent scientists and patients working together, and can be downloaded free online (declaration: I wrote the foreword!).

Testing Treatments (free online)

http://www.jameslindlibrary.org/testing-treatments.html

My book “Bad Science” has a lot on this topic, and my next book “Bad Pharma” has more. There is also lots on the problem of publication bias at my website:

www.badscience.net

What’s the solution?

We need to publish all trials, for all drugs, conducted in humans.

What can you do?

There are some things I think you could do, if this situation worries you:

1. Shout about it.

2. Tell people it’s not been fixed

3. Lobby for change.

More reading - and a lobbying toolkit - will be available shortly at:

www.alltrials.org