Turning a blind eye: the success of blinding reported in a random sample of randomised, placebo controlled trials
http://www.100md.com
《英国医生杂志》
1 Ottawa Health Research Institute, Clinical Epidemiology Program, 501 Smyth Road, Box 201, Ottawa, ON, Canada K1H 8L6, 2 Departments of Human Genetics & Pediatrics and Biomedical Ethics Unit, McGill University, Montreal, QC, Canada, 3 Department of Epidemiology and Biostatistics, McGill University, 4 Research Ethics and Regulation Group, Faculty of Law, University of Toronto, Toronto, ON, Canada
Correspondence to: D Fergusson dafergusson@ohri.ca
Abstract
Although the definition of double blind varies,1 we consider a trial to be double blind when the patient, investigators, and outcome assessors are unaware of the patient's assigned treatment throughout the conduct of the trial.2 Placebos are commonly used as an inactive treatment to achieve double blinding. Active placebos, with which symptoms or side effects are imitated, can also be used. Placebos are justly used when no existing effective treatment is available. Sometimes, placebos are proposed instead of a standard existing treatment or standard care to ensure assay sensitivity. That is, to demonstrate the effectiveness of a new treatment, it must be demonstrated against a "clean" control. The argument is that although the new treatment may be found to be as effective or more effective than standard treatment in a clinical trial, both treatments may very well be ineffective. Assay sensitivity is the ability of a trial to distinguish effective interventions from ineffective interventions. It depends on the effect size that is to be detected. As such, the investigators need to know the anticipated effects of the control intervention. It is argued that placebos are the ideal choice as their anticipated benefits are known to be marginal. This argument is predicated on the belief that participants, investigators, and outcome assessors remain blinded to the treatment assignment. If the blinding of the placebo arm is not effective then the protection against expectation effects, biased assessment, contamination, and co-intervention are all lost. The observed superiority of a new treatment over placebo could merely be a consequence of loss of this control—and an ineffective new treatment would spuriously seem to be superior. Because of the importance of the success of blinding, the Consolidated Standards for Reporting of Trials (CONSORT) Group has explicitly incorporated the issue. Section 11(b) of the CONSORT statement states that the success of blinding is to be reported in the publication.3
It is not sufficient that trials describe themselves as double blind. It is also important that the efficacy of the blinding is actually assessed. In other words, an assessment of the face validity of the double blinding is needed. To assess the reporting and success of double blinding, we chose a random sample of randomised, placebo controlled trials from leading journals in general medicine and psychiatry. Although we have focused on placebo controlled trials, the issues discussed also arise in double blind trials with active controls.
Methods
The quality of reporting in clinical trials has evolved. Over the years, trialists have been held more accountable and responsible for the quality of trial reporting. This evolution began with the need for reporting the numbers of patients screened, enrolled, randomised, and analysed,19 and progressed to the reporting of patient withdrawals and its importance for the analysis and interpretation of study results.20 Building on this progress, there is a need for trialists and journals routinely to report the methods of blinding and the subsequent success of this blinding.21
Our examination of the success of blinding challenges the notion that placebo controlled trials inherently possess assay sensitivity. Clearly, there is a failure among investigators and journals in reporting the success of blinding. Only 15 of the 191 trials (8%) provided such information, be it qualitative or quantitative. Of the 15 trials, only five trials reported that blinding was successful,9 12 13 16 17 and of these, three did not present any quantitative data analysis to support their claim.9 13 16
Only four trials assessed blinding in both the participants and either the outcome assessors or the investigators.6 12 16 17 Thus, the face validity of the double blinding was only reported in four of the 191 articles (2%). This deficiency in reporting translates into a paucity of evidence that a placebo ensures a "clean" control. Furthermore, the quality of evidence in the few studies that reported on the success of blinding is weak on two fronts: the quality of the data and the evidence that blinding was successful. The success of blinding was described as less than optimal in nine of the 14 trials that reported on blinding, and of the five trials that reported that blinding was maintained, only two provided data to support their claim.12 17 Unfortunately, when we examined the data and analysis provided by these two trials we found that their claim of success is debatable.
We would like to see Item 11b of CONSORT revised to require the assessment of blinding for all double blind randomised trials. Trialists have an ethical responsibility to justify the use of a placebo for blinding purposes in their research protocol and informed consent procedures. Thus, it seems reasonable to suggest that an assessment of the success of blinding is necessary. If blinding is not assessed, we may delude ourselves as to exactly what information we gain from incorporating a placebo comparison. Although all trials should assess blinding, the types of trials that will particularly benefit are trials with subjective outcomes or outcomes reported by patients (for example, quality of life instruments), or trials where the side effects are well known. Even though there may be problems with analysing and interpreting the results of success, this does not provide a rationale for not doing it. Clearly, the lack of successful blinding can bias observed estimates of effect. Although this bias is differential, its direction may not be easily ascertained. We might anticipate that evidence of unsuccessful blinding in a "double blind" active versus placebo trial would result in a positive bias and hence lead to an overestimate of the treatment effect. However, unblinded patients receiving placebo may seek other treatments, especially if there is established effective treatment available, and this makes the extent and even the direction of bias difficult to determine.
We believe that trialists need to report a minimum set of information. This includes the counts of all patients allocated to each treatment; the counts of patients who guess treatment assignment by the group to which they were allocated; the counts of correct guesses and those who are undecided; the analytical methods and results used to assess success of blinding; and the author's interpretation of the efficacy of blinding and the effect on study results. The data abstracted for this study show a substantial lack of reporting with respect to these minimum, essential items, as illustrated by the number of vacant fields in tables 3 and 4.
What is already known on this topic
Placebo controls are commonly used in randomised trials to blind investigators, outcome assessors, and patients to treatment assignment
Placebo controls have been advocated instead of existing effective treatment because they ensure assay sensitivity
Unsuccessful double blinding results in a differential bias of effect measures
What this study adds
The success of blinding is not well reported
The success of blinding in trials that do report is often poor
Little evidence exists that placebos provide assay sensitivity
The current lack of reporting on the success of blinding provides little evidence that success of blinding is maintained in placebo controlled trials. Trialists and editors need to make a concerted effort to incorporate, report, and publish such information and its potential effect on study results. The efficacy of the blinding cannot be assumed on theoretical grounds. We need evidence before we can assert that assay sensitivity exists in randomised, double blind, placebo controlled trials.
Amendment
This is Version 2 of the paper. In this version, the references in tables 3 and 4 have been corrected. They now start with reference 4 and end with reference 18 .
We thank Julie Comber and Jennifer Marshall for article retrieval and data collection.
Contributors: DF, KG, DW, and SS conceived and designed the study. DF collected, managed, and analysed the data. All authors interpreted the data and wrote the paper. DF is the guarantor.
Funding: This work was funded in part by the Canadian Institutes of Health Research.
Competing interests: None declared.
Ethical approval: Not required.
References
Devereaux PJ, Manns BJ, Ghali WA, Quan H, Lacchetti C, Montori VM, et al. Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA 2001;285: 2000-3.
Schulz KF, Chalmers I, Altman DG. The landscape and lexicon of blinding in randomized trials. Ann Intern Med. 2002;136: 254-9.
Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001;134: 663-94.
Sackeim HA, Haskett RF, Mulsant BH, Thase ME, Mann JJ, Pettinati HM, et al. Continuation pharmacotherapy in the prevention of relapse following electroconvulsive therapy: a randomized controlled trial. JAMA 2001;285: 1299-307.
Rowe PC, Calkins H, DeBusk K, McKenzie R, Anand R, Sharma G, et al. Fludrocortisone acetate to treat neurally mediated hypotension in chronic fatigue syndrome: a randomized controlled trial. JAMA 2001;285: 52-9.
Apfel SC, Schwartz S, Adornato BT, Freeman R, Biton V, Rendell M, et al. Efficacy and safety of recombinant human nerve growth factor in patients with diabetic polyneuropathy: a randomized controlled trial. rhNGF Clinical Investigator Group. JAMA 2000;284: 2215-21.
Von Schacky C, Angerer P, Kothny W, Theisen K, Mudra H. The effect of dietary omega-3 fatty acids on coronary atherosclerosis. A randomised, double-blind, placebo-controlled trial. Ann Intern Med 1999;130: 554-62.
Sandler RS, Zorich NL, Filloon TG, Wiseman HB, Lietz DJ, Brock MH, et al. Gastrointestinal symptoms in 3181 volunteers ingesting snack foods containing olestra or triglycerides. A 6-week randomised, placebo-controlled trial. Ann Intern Med 1999;130: 253-61.
Blondal T, Gudmundsson LJ, Olafsdottir I, Gustavsson G, Westin A. Nicotine nasal spray with nicotine patch for smoking cessation: randomised trial with six year follow up. BMJ 1999;318: 285-8.
Shlay JC, Chaloner K, Max MB, Flaws B, Reichelderfer P, Wentworth D, et al. Acupuncture and amitriptyline for pain due to HIV-related peripheral neuropathy: a randomised controlled trial. Terry Beirn Community Programs for Clinical Research on AIDS. JAMA 1998;280: 1590-5.
Wisner KL, Perel JM, Peindl KS, Hanusa BH, Findling RL, Rapport D. Prevention of recurrent postpartum depression: a randomised clinical trial. J Clin Psychiatry 2001;62: 82-6.
Warner J, Metcalfe C, King M. Evaluating the use of benzodiazepines following recent bereavement. Br J Psychiatry 2001;178: 36-41.
Ben Zion IZ, Meiri G, Greenberg BD, Murphy DL, Benjamin J. Enhancement of CO2-induced anxiety in healthy volunteers with the serotonin antagonist metergoline. Am J Psychiatry 1999;156: 1635-7.
Himle JA, Abelson JL, Haghightgou H, Hill EM, Nesse RM, Curtis GC. Effect of alcohol on social phobic anxiety. Am J Psychiatry 1999;156: 1237-43.
Stoll AL, Severus WE, Freeman MP, Rueter S, Zboyan HA, Diamond E, et al. Omega 3 fatty acids in bipolar disorder: a preliminary double-blind, placebo-controlled trial. Arch Gen Psychiatry 1999;56: 407-12.
Heresco-Levy U, Javitt DC, Ermilov M, Mordel C, Silipo G, Lichtenstein M. Efficacy of high-dose glycine in the treatment of enduring negative symptoms of schizophrenia. Arch Gen Psychiatry 1999;56: 29-36.
Schneier FR, Goetz D, Campeas R, Fallon B, Marshall R, Liebowitz MR. Placebo-controlled trial of moclobemide in social phobia. Br J Psychiatry 1998;172: 70-7.
Young SA, Hurt PH, Benedek DM, Howard RS. Treatment of premenstrual dysphoric disorder with sertraline during the luteal phase: a randomised, double-blind, placebo-controlled crossover trial. J Clin Psychiatry 1998;59: 76-80.
Sackett DL, Gent M. Controversy in counting and attributing events in clinical trials. N Engl J Med 1979;301: 1410-2.
Sheiner LB, Rubin DB. Intention-to-treat analysis and the goals of clinical trials. Clin Pharmacol Ther 1995;57: 6-15.
Schulz KF, Grimes DA, Altman DG, Hayes RJ. Blinding and exclusions after allocation in randomised controlled trials: survey of published parallel group trials in obstetrics and gynaecology. BMJ 1996;312: 742-4.(Dean Fergusson, scientist)
Correspondence to: D Fergusson dafergusson@ohri.ca
Abstract
Although the definition of double blind varies,1 we consider a trial to be double blind when the patient, investigators, and outcome assessors are unaware of the patient's assigned treatment throughout the conduct of the trial.2 Placebos are commonly used as an inactive treatment to achieve double blinding. Active placebos, with which symptoms or side effects are imitated, can also be used. Placebos are justly used when no existing effective treatment is available. Sometimes, placebos are proposed instead of a standard existing treatment or standard care to ensure assay sensitivity. That is, to demonstrate the effectiveness of a new treatment, it must be demonstrated against a "clean" control. The argument is that although the new treatment may be found to be as effective or more effective than standard treatment in a clinical trial, both treatments may very well be ineffective. Assay sensitivity is the ability of a trial to distinguish effective interventions from ineffective interventions. It depends on the effect size that is to be detected. As such, the investigators need to know the anticipated effects of the control intervention. It is argued that placebos are the ideal choice as their anticipated benefits are known to be marginal. This argument is predicated on the belief that participants, investigators, and outcome assessors remain blinded to the treatment assignment. If the blinding of the placebo arm is not effective then the protection against expectation effects, biased assessment, contamination, and co-intervention are all lost. The observed superiority of a new treatment over placebo could merely be a consequence of loss of this control—and an ineffective new treatment would spuriously seem to be superior. Because of the importance of the success of blinding, the Consolidated Standards for Reporting of Trials (CONSORT) Group has explicitly incorporated the issue. Section 11(b) of the CONSORT statement states that the success of blinding is to be reported in the publication.3
It is not sufficient that trials describe themselves as double blind. It is also important that the efficacy of the blinding is actually assessed. In other words, an assessment of the face validity of the double blinding is needed. To assess the reporting and success of double blinding, we chose a random sample of randomised, placebo controlled trials from leading journals in general medicine and psychiatry. Although we have focused on placebo controlled trials, the issues discussed also arise in double blind trials with active controls.
Methods
The quality of reporting in clinical trials has evolved. Over the years, trialists have been held more accountable and responsible for the quality of trial reporting. This evolution began with the need for reporting the numbers of patients screened, enrolled, randomised, and analysed,19 and progressed to the reporting of patient withdrawals and its importance for the analysis and interpretation of study results.20 Building on this progress, there is a need for trialists and journals routinely to report the methods of blinding and the subsequent success of this blinding.21
Our examination of the success of blinding challenges the notion that placebo controlled trials inherently possess assay sensitivity. Clearly, there is a failure among investigators and journals in reporting the success of blinding. Only 15 of the 191 trials (8%) provided such information, be it qualitative or quantitative. Of the 15 trials, only five trials reported that blinding was successful,9 12 13 16 17 and of these, three did not present any quantitative data analysis to support their claim.9 13 16
Only four trials assessed blinding in both the participants and either the outcome assessors or the investigators.6 12 16 17 Thus, the face validity of the double blinding was only reported in four of the 191 articles (2%). This deficiency in reporting translates into a paucity of evidence that a placebo ensures a "clean" control. Furthermore, the quality of evidence in the few studies that reported on the success of blinding is weak on two fronts: the quality of the data and the evidence that blinding was successful. The success of blinding was described as less than optimal in nine of the 14 trials that reported on blinding, and of the five trials that reported that blinding was maintained, only two provided data to support their claim.12 17 Unfortunately, when we examined the data and analysis provided by these two trials we found that their claim of success is debatable.
We would like to see Item 11b of CONSORT revised to require the assessment of blinding for all double blind randomised trials. Trialists have an ethical responsibility to justify the use of a placebo for blinding purposes in their research protocol and informed consent procedures. Thus, it seems reasonable to suggest that an assessment of the success of blinding is necessary. If blinding is not assessed, we may delude ourselves as to exactly what information we gain from incorporating a placebo comparison. Although all trials should assess blinding, the types of trials that will particularly benefit are trials with subjective outcomes or outcomes reported by patients (for example, quality of life instruments), or trials where the side effects are well known. Even though there may be problems with analysing and interpreting the results of success, this does not provide a rationale for not doing it. Clearly, the lack of successful blinding can bias observed estimates of effect. Although this bias is differential, its direction may not be easily ascertained. We might anticipate that evidence of unsuccessful blinding in a "double blind" active versus placebo trial would result in a positive bias and hence lead to an overestimate of the treatment effect. However, unblinded patients receiving placebo may seek other treatments, especially if there is established effective treatment available, and this makes the extent and even the direction of bias difficult to determine.
We believe that trialists need to report a minimum set of information. This includes the counts of all patients allocated to each treatment; the counts of patients who guess treatment assignment by the group to which they were allocated; the counts of correct guesses and those who are undecided; the analytical methods and results used to assess success of blinding; and the author's interpretation of the efficacy of blinding and the effect on study results. The data abstracted for this study show a substantial lack of reporting with respect to these minimum, essential items, as illustrated by the number of vacant fields in tables 3 and 4.
What is already known on this topic
Placebo controls are commonly used in randomised trials to blind investigators, outcome assessors, and patients to treatment assignment
Placebo controls have been advocated instead of existing effective treatment because they ensure assay sensitivity
Unsuccessful double blinding results in a differential bias of effect measures
What this study adds
The success of blinding is not well reported
The success of blinding in trials that do report is often poor
Little evidence exists that placebos provide assay sensitivity
The current lack of reporting on the success of blinding provides little evidence that success of blinding is maintained in placebo controlled trials. Trialists and editors need to make a concerted effort to incorporate, report, and publish such information and its potential effect on study results. The efficacy of the blinding cannot be assumed on theoretical grounds. We need evidence before we can assert that assay sensitivity exists in randomised, double blind, placebo controlled trials.
Amendment
This is Version 2 of the paper. In this version, the references in tables 3 and 4 have been corrected. They now start with reference 4 and end with reference 18 .
We thank Julie Comber and Jennifer Marshall for article retrieval and data collection.
Contributors: DF, KG, DW, and SS conceived and designed the study. DF collected, managed, and analysed the data. All authors interpreted the data and wrote the paper. DF is the guarantor.
Funding: This work was funded in part by the Canadian Institutes of Health Research.
Competing interests: None declared.
Ethical approval: Not required.
References
Devereaux PJ, Manns BJ, Ghali WA, Quan H, Lacchetti C, Montori VM, et al. Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA 2001;285: 2000-3.
Schulz KF, Chalmers I, Altman DG. The landscape and lexicon of blinding in randomized trials. Ann Intern Med. 2002;136: 254-9.
Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001;134: 663-94.
Sackeim HA, Haskett RF, Mulsant BH, Thase ME, Mann JJ, Pettinati HM, et al. Continuation pharmacotherapy in the prevention of relapse following electroconvulsive therapy: a randomized controlled trial. JAMA 2001;285: 1299-307.
Rowe PC, Calkins H, DeBusk K, McKenzie R, Anand R, Sharma G, et al. Fludrocortisone acetate to treat neurally mediated hypotension in chronic fatigue syndrome: a randomized controlled trial. JAMA 2001;285: 52-9.
Apfel SC, Schwartz S, Adornato BT, Freeman R, Biton V, Rendell M, et al. Efficacy and safety of recombinant human nerve growth factor in patients with diabetic polyneuropathy: a randomized controlled trial. rhNGF Clinical Investigator Group. JAMA 2000;284: 2215-21.
Von Schacky C, Angerer P, Kothny W, Theisen K, Mudra H. The effect of dietary omega-3 fatty acids on coronary atherosclerosis. A randomised, double-blind, placebo-controlled trial. Ann Intern Med 1999;130: 554-62.
Sandler RS, Zorich NL, Filloon TG, Wiseman HB, Lietz DJ, Brock MH, et al. Gastrointestinal symptoms in 3181 volunteers ingesting snack foods containing olestra or triglycerides. A 6-week randomised, placebo-controlled trial. Ann Intern Med 1999;130: 253-61.
Blondal T, Gudmundsson LJ, Olafsdottir I, Gustavsson G, Westin A. Nicotine nasal spray with nicotine patch for smoking cessation: randomised trial with six year follow up. BMJ 1999;318: 285-8.
Shlay JC, Chaloner K, Max MB, Flaws B, Reichelderfer P, Wentworth D, et al. Acupuncture and amitriptyline for pain due to HIV-related peripheral neuropathy: a randomised controlled trial. Terry Beirn Community Programs for Clinical Research on AIDS. JAMA 1998;280: 1590-5.
Wisner KL, Perel JM, Peindl KS, Hanusa BH, Findling RL, Rapport D. Prevention of recurrent postpartum depression: a randomised clinical trial. J Clin Psychiatry 2001;62: 82-6.
Warner J, Metcalfe C, King M. Evaluating the use of benzodiazepines following recent bereavement. Br J Psychiatry 2001;178: 36-41.
Ben Zion IZ, Meiri G, Greenberg BD, Murphy DL, Benjamin J. Enhancement of CO2-induced anxiety in healthy volunteers with the serotonin antagonist metergoline. Am J Psychiatry 1999;156: 1635-7.
Himle JA, Abelson JL, Haghightgou H, Hill EM, Nesse RM, Curtis GC. Effect of alcohol on social phobic anxiety. Am J Psychiatry 1999;156: 1237-43.
Stoll AL, Severus WE, Freeman MP, Rueter S, Zboyan HA, Diamond E, et al. Omega 3 fatty acids in bipolar disorder: a preliminary double-blind, placebo-controlled trial. Arch Gen Psychiatry 1999;56: 407-12.
Heresco-Levy U, Javitt DC, Ermilov M, Mordel C, Silipo G, Lichtenstein M. Efficacy of high-dose glycine in the treatment of enduring negative symptoms of schizophrenia. Arch Gen Psychiatry 1999;56: 29-36.
Schneier FR, Goetz D, Campeas R, Fallon B, Marshall R, Liebowitz MR. Placebo-controlled trial of moclobemide in social phobia. Br J Psychiatry 1998;172: 70-7.
Young SA, Hurt PH, Benedek DM, Howard RS. Treatment of premenstrual dysphoric disorder with sertraline during the luteal phase: a randomised, double-blind, placebo-controlled crossover trial. J Clin Psychiatry 1998;59: 76-80.
Sackett DL, Gent M. Controversy in counting and attributing events in clinical trials. N Engl J Med 1979;301: 1410-2.
Sheiner LB, Rubin DB. Intention-to-treat analysis and the goals of clinical trials. Clin Pharmacol Ther 1995;57: 6-15.
Schulz KF, Grimes DA, Altman DG, Hayes RJ. Blinding and exclusions after allocation in randomised controlled trials: survey of published parallel group trials in obstetrics and gynaecology. BMJ 1996;312: 742-4.(Dean Fergusson, scientist)