Universiteit Utrecht

Department of Mathematics

Careless Statistics Cost Lives: the Utrecht probiotica trial

Richard Gill
(Leiden University)

7 May 2008

A randomized clinical trial comparing an experimental new treatment to a standard therapy for a life-threatening medical condition should be stopped early on ethical grounds, in either of the following scenarios: (1) It has become overwhelmingly clear that the new treatment is better than the standard. (2) It has become overwhelmingly clear that the trial is not going to show that the new treatment is any better than the standard. The trial is continued in the third scenario: (3) There is a reasonable chance that the new treatment will finally turn out to be better than the standard, but we aren't sure yet. However, the (blinded) data monitoring committee in the "PROPATRIA" trial of an experimental probiotica treatment for patients with acute pancreatitis allowed the trial to continue at the half way interim analysis, in effect because there was still a good chance of proving that the probiotica treatment was very harmful to their patients. The committee did not know whether treatment A was the probiotica treatment or the placebo. In itself this should not have caused a problem, since it could easily have determined the appropriate decision under both scenarios. Were the decisions in the two scenarios different, then the data would have to be de-blinded, in order to determine the appropriate decision. And the decisions in this trial were indeed different: were A the probiotica treatment, the monitoring committee would have found themselves in scenario (2) and would have stopped the trial; were A the placebo, the monitoring committee would have found themselves in scenario (3) and would have let the trial continue. The PROPATRIA team used a two-sided version due to Schouten (1999, standard Dutch textbook "Klinische Statistiek") of a one-sided early- stopping-rule of Snapinn (1992, Statistics in Medicine). However, they only refer to the Snapinn publication, making it very difficult to work out what they actually did. In particular, there is no way that international referees could check the protocol being used by the team. Schouten's two-sided variant is appropriate when we are comparing two existing standard treatments, in which case the ethical issues of early stopping are quite different. Actually, only now that Schouten's book is identified as being the main resource of the team, does it become clear that the PROPATRIA team were not only confused about the difference between one- and two-sided statistical testing, but also between errors of the first and second kinds. Referring to Snapinn, they quote table values from Schouten which are actually taken from the wrong column of the wrong table (even if we were to accept the use of a two-sided test). But the mistake which had fatal consequences was to use Schouten's two- sided early-stopping-rule in conjunction with the blindedness of the data monitoring committee. This rule does not distinguish between continuing the trial because there is hope to prove that A is better than B, and continuing the trail because there is hope to prove that B is better than A. It seems that about 5 lives were sacrificed to the chance of getting a significant result that the probiotica treatment was bad for the patients in the trial.

Back to the history of the seminar or the Colloquium Stochastiek homepage.
Charlene Kalle