Systematic review to determine whether participation in a trial influe
http://www.100md.com
《英国医生杂志》
1 Norwegian Health Services Research Centre, PO Box 7004, 0130 Oslo, Norway, 2 National Research Centre for Rehabilitation in Rheumatology, Diakonhjemmet Hospital, Oslo, 3 Clinical Epidemiology and Biostatistics, McMaster University Health Sciences Centre, Hamilton, Ontario, Canada
Correspondence to: G E Vist gev@nhsrc.no
Objective To systematically compare the outcomes of participants in randomised controlled trials (RCTs) with those in comparable non-participants who received the same or similar treatment.
Table 1 Summary of studies included in sensitivity analysis
Data sources Bibliographic databases, reference lists from eligible articles, medical journals, and study authors.
Review methods RCTs and cohort studies that evaluated the clinical outcomes of participants in RCTs and comparable non-participants who received the same or similar treatment.
Results Five RCTs (six comparisons) and 50 cohort studies (85 comparisons) provided data on 31 140 patients treated in RCTs and 20 380 comparable patients treated outside RCTs. In the five RCTs, in which patients were given the option of participating or not, the comparisons provided limited information because of small sample sizes (a total of 412 patients) and the nature of the questions considered. 73 dichotomous outcomes were compared, of which 59 reported no statistically significant differences. For patients treated within RCTs, 10 comparisons reported significantly better outcomes and four reported significantly worse outcomes. Significantly heterogeneity was found (I2 = 89%) among the comparisons of 73 dichotomous outcomes; none of our a priori explanatory factors helped explain this heterogeneity. The 18 comparisons of continuous outcomes showed no significant differences in heterogeneity (I2 = 0%). The overall pooled estimate for continuous outcomes of the effect of participating in an RCT was not significant (standardised mean difference 0.01, 95% confidence interval -0.10 to 0.12).
Conclusion No strong evidence was found of a harmful or beneficial effect of participating in RCTs compared with receiving the same or similar treatment outside such trials.
Properly conducted randomised controlled trials (RCTs) provide the strongest evidence of the effects of treatment.1 It is, however, controversial as to whether participants of such trials benefit directly or whether these studies are solely for the benefit of future patients. In addition, there is much scepticism about the applicability of the results to usual practice.2
Four reviews that considered whether it is beneficial or harmful to participate in RCTs drew varied conclusions.3-6 These reviews compared patients who were treated within trials with those treated outside the trials, regardless of differences between the clinical interventions or between the participants and non-participants. It is therefore uncertain whether the results reflect the effects of participating in an RCT (trial effects), differences in the clinical interventions (treatment effects), or differences between participants and non-participants. We determined whether the outcomes of participants in RCTs differed from those of comparable non-participants who received the same or similar treatment.
Methods
Our review was undertaken as a Cochrane methodology review (see Cochrane Library for fuller details of our methods and updated versions of the review).7
We included studies that compared participants in RCTs with comparable non-participants who received the same or similar treatment. We included observational studies and RCTs in which participation or the option of participation was randomly allocated.
Search strategies
We used seven strategies to identify relevant studies: consultation with experts; search of personal files; electronic searches of the Cochrane central register of controlled trials, Medline, Embase, the Cochrane methodology register, and PsycINFO; a review of reference lists from eligible articles; and a search of PubMed using the "related articles" feature and SciSearch. We also hand searched articles published in 2000 in five medical journals (BMJ, Annals of Internal Medicine, JAMA, Lancet, and New England Journal of Medicine) to identify RCTs with over 200 patients and at least 100 eligible non-participants. Studies' authors were contacted for data on the treatment and outcomes of eligible non-participants.
Assessment of study eligibility
Two reviewers independently assessed each article for eligibility. Disagreements were resolved by discussion. A third reviewer was consulted when consensus could not be reached.
Data ion
Two reviewers independently ed data from eligible studies. Each study was assessed for selection bias (differences between participants and non-participants), detection bias, and exclusion bias (losses to follow-up). On the basis of the combined risks of the three biases, we grouped each comparison into overall quality groups (randomised, controlled comparisons, partially controlled comparisons, and poorly controlled comparisons) for analysis. Missing data were sought from the investigators.
Analysis
We compared the experimental group of the RCT with their respective eligible non-participants who received the treatment and the control group and eligible non-participants who received the control treatment outside of the trial. For each comparison, we analysed the main outcome, and, when reported, we analysed mortality separately. For all of the included comparisons we used the main outcome as reported by the investigators. We analysed the dichotomous and continuous results separately. The results are reported as relative risks with 95% confidence intervals, using adjusted estimates when available. For a summary of the dichotomous results in one table, we calculated the associated standard error from the natural logarithm of the unadjusted relative risk. Heterogeneity was assessed by 2 test and the I2 statistic using RevMan version 4.2.8 For the unadjusted relative risk analysis, we used the Mantel-Haenszel test in RevMan. A fixed effect model was used to calculate summary statistics if no statistically significant (P < 0.10) heterogeneity was found among similar comparisons. For statistically heterogeneous results we described the variation in the estimates and key explanatory factors; where possible relating the explanatory factors to observed differences in estimates of the effects of participation. We constructed a funnel plot to explore the possibility of publication bias.
Fig 1 Selection process of eligible studies
Results
Overall, 55 studies, totalling 91 comparisons, met our inclusion criteria (fig 1). Forty one studies are still awaiting assessment, which currently cannot be included or excluded on the basis of the published information.
We identified five RCTs (six comparisons) in which patients were randomised according to whether they had the option to participate. These studies provided limited data because of their small sample sizes and the nature of the questions considered. Two studies randomised 82 patients to "n of 1 trials" compared with standard practice—that is, randomised, double blind, multiple crossover comparisons of an active drug with a placebo in a single patient.9 10 One study (60 patients) measured spontaneously self reported side effects in patients who had or had not been informed that they were in an RCT.11 One study (227 patients) reported satisfaction among patients randomised to an RCT compared with patients randomised to a patient preference trial in which they had a choice of treatment.12 Another study (43 patients) reported pain reduction among patients randomised to an RCT compared with those who were not invited to participate.13 None of these studies found significant differences in outcomes between patients treated in or outside RCTs.
Non-randomised cohort studies
We identified 50 cohort studies (85 comparisons) totalling 30 862 patients participating in RCTs compared with 20 246 patients treated outside RCTs.14-63 Seventy comparisons comprised dichotomous outcomes, of which 12 reported adjusted estimates, and 15 comparisons comprised continuous outcomes.
We found significant heterogeneity (I2 = 89.0%) among the results of comparisons with dichotomous main outcomes (fig 2); these results were therefore not pooled. Of these 73 comparisons, 59 reported no significant differences between outcomes for patients treated in RCTs and those receiving similar treatments outside RCTs; 10 reported significantly better outcomes for patients treated in RCTs, and four reported significantly worse outcomes for patients treated in RCTs.
Fig 2 Results of dichotomous main outcomes in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
Figure 3 shows the results of the 18 comparisons with continuous main outcomes. We found no significant heterogeneity (I2 = 0%). The pooled estimate found no differences in outcomes for patients treated in and outside RCTs (standardised mean difference 0.01, 95% confidence interval -0.10 to 0.12).
Fig 3 Results of continuous main outcomes in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
In 17 studies (32 comparisons) with data on mortality (fig 4), we found significant heterogeneity (I2 = 88.8%); the results were therefore not pooled. In 24 of the 32 comparisons we found no significant difference in mortality. Four comparisons reported a significant lower risk of mortality for patients treated in RCTs and four comparisons reported a significantly higher risk of mortality.
Fig 4 Comparison of mortality between participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
Separate subgroup analyses could not explain the observed heterogeneity by the different types of eligible non-participants, treatments, clinical specialities, or study quality (selection bias, detection bias, and exclusion bias). The table summarises the studies included in the sensitivity analysis.
The funnel plot of the dichotomous comparisons showed no asymmetry (fig 5), indicating a low risk of publication bias.
Fig 5 Funnel plot of dichotomous comparisons in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
Discussion
Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane methodology review). Issue 4. Oxford: Update Software, 2002.
Rothwell PM. External validity of randomised controlled trials: "To whom do the results of this trial apply?" Lancet 2005;365: 82-93.
Stiller CA. Centralised treatment, entry to trials and survival. Br J Cancer 1994;70: 352-62.
Braunholtz DA, Edwards SJL, Lilford RJ. Are randomized clinical trials good for us (in the short term)? Evidence for a "trial effect." J Clin Epidemiol 2001;54: 217-24.
The Emergency Care Research Institute 2002. Patients' reasons for participation in clinical trials and effect of trial participation on patient outcomes. www.ecri.org/Patient_Information/Patient_Reference_Guide/evidence.pdf (accessed Nov 2004).
Peppercorn JM, Weeks JC, Cook EFC, Joffe S. Comparison of outcomes in cancer patients treated within and outside clinical trials: conceptual framework and structured review. Lancet 2004;363: 263-70.
Vist GE, Hagen KB, Devereaux P, Bryant D, Kristoffersen DT, Oxman AD. Outcomes of patients who participate in randomised controlled trials compared to similar patients receiving similar interventions who do not participate (Cochrane methodology reviews). Issue 1. Oxford: Update Software, 2005.
Higgins JP, Thompson SG. Quantifying heterogeneity in meta-analysis. Stat Med 2002;21: 1539-58.
Mahon J, Laupacis A, Donner A, Wood T. Randomised study of n of 1 trials versus standard practice. BMJ 1996;312: 1069-74.
Mahon JL, Laupacis A, Hodder RV, McKim DA, Paterson NAM, Wood TE, et al. Theophylline for irreversible chronic airflow limitation. A randomized study comparing n of 1 trials to standard practice. Chest 1999;115: 38-48.
Dahan R, Caulin C, Figea L, Kanis JA, Caulin F, Segrestaa JM. Does informed consent influence therapeutic outcome? A clinical trial of the hypnotic activity of placebo in patients admitted to hospital. BMJ 1986;293: 363-4.
Cooper KG, Grant AM, Garratt AM. The impact of using a partially randomised patient preference design when evaluating alternative managements for heavy menstrual bleeding. Br J Obstet Gynaecol 1997;104: 1367-73.
Bergmann JF, Chassany O, Gandiol J, Deblois P, Kanis JA, Segrestaa JM, et al. A randomised clinical trial of the effect of informed consent on the analgesic activity of placebo and naproxen in cancer pain. Clin Trials Metaanal 1994;29: 41-7.
Balmukhanov SB, Beisebaev AA, Aitkoolova ZI, Mustaphin JS, Philippenko VI, Rismuhamedova RS, et al. Intramural and parametrial infusion of metronidazole in the radiotherapy of uterine cervix cancer: preliminary report. Int J Radiation Oncol Biol Phys 1989;16: 1061-3.
Baum E, Sather H, Nachman J, Seinfeld J, Kritvit W, Leikin S, et al. Relapse rates following cessation of chemotherapy during complete remission of acute lymphocytic leukemia: a report from childrens cancer study group. Med Pediatr Oncol 1979;7: 25-34.
Bedi N, Chilvers C, Churchill R, Dewey M, Duggan C, Fielding K, et al. Assessing effectiveness of treatment of depression in primary care: partially randomised preference trial. Br J Psychol 2000;177: 312-8.
Berglund G, Bolund C, Gustafsson UL, Sjoden PO. Is the wish to participate in a cancer rehabilitation program an indicator of the need? Comparisons of participants and non-participants in a randomized study. Psychooncology 1997;6: 35-46.
Bhattacharya S, Cameron IM, Mollison J, Parkin DE, Abramovich DR, Kitchener HC. Admission-discharge policies for hysteroscopic surgery: a randomised comparison of day case with in-patient admission. Eur J Obstet Gynecol Reprod Biol 1998;76: 81-4.
Bijker N, Peterse JL, Fentiman TS, Julien JP, Hart AAM, Avril A, et al. Effects of patient selection on the applicability of results from a randomised clinical trial (EORTC 10853) investigating breast-conserving treatment for DCIS. Breast conserving therapy for ductal carcinoma in situ. PhD Thesis, 2000.
Blichert-Toft M, Brincker H, Andersen JA, Andresen KW, Axelsson CK, Mouridsen HT, et al.. A Danish randomized trial comparing breast-conserving therapy with mastectomy in mammary carcinoma: preliminary results. Acta Oncologica 1988;27: 671-7.
Coronary artery surgery study (CASS): a randomised trial of coronary artery bypass surgery. Comparability of entry characteristics and survival in randomized patients and nonrandomized patients meeting randomization criteria. J Am Coll Cardiol 1984;3: 114-28.
Chauhan SP, Rutherford SE, Hess LW, Morrison JC. Prophylactic intrapartum amnioinfusion for patients with oligohydramnios: a prospective randomized study. J Reprod Med 1991;37: 817-20.
Chilvers C, Dewey M, Fielding K, Gretton V, Miller P, Palmer B, et al. Antidepressant drugs and generic counselling for treatment of major depression in primary care: randomised trial with patient preference arms. BMJ 2001;322: 772-5.
Clagett GP, Youkey JR, Bringham RA, Orecchia PM, Salander JM, Collins GJ, et al. Asymptomatic cervical bruit and abnormal ocular pneumoplethysmography: a prospective study comparing two approaches to management. Surgery 1984;96: 823-30.
Creutzig U, Ritter J, Zimmermann M, Schellong G. Does cranial irradiation reduce the risk of bone marrow relapse in acute myelogenous leukemia? Unexpected results of the childhood acute myelogenous leukemia study BFM-87. J Clin Oncol 1993;11: 279-86.
Davis S, Wright PW, Schulman SF, Hill LD, Pinkham RD, Johnson LP, et al. Participants in prospective, randomized clinical trials for resected non-small cell lung cancer have improved survival compared with nonparticipants in such trials. Cancer 1985;56: 1710-8.
Edsmyr F, Esposti PL, Johansson B, Strindberg B. Clinical experimental randomized study of 2.6-CIS-diphenylhexamethylcyclotetrasiloxane and estramurine-17-phosphate in the treatment of prostatic carcinoma. J Urol 1978;120: 705-7.
Feit F, Brooks MM, Sopko G, Keller NM, Rosen A, Krone R, et al. Long-term clinical outcome in the bypass angioplasty revascularization investigation registry: comparison with the randomized trial. Circulation 2000;101: 2795-802.
Forbes GM, Collins BJ. Nitrous oxide for colonoscopy: a randomized controlled study. Gastrointest Endosc 2000;51: 271-7.
Forssell C, Takolander R, Bergqvist D, Johansson A, Persson NH. Local versus general anaesthesia in carotid surgery. A prospective, randomised study. Eur J Vasc Surg 1989;3: 503-9.
Helsing M, Bergman B, Thaning L, Hero U, for the Joint Lung Cancer Study Group. Quality of life and survival in patients with advanced non-small cell lung cancer receiving supportive care plus chemotherapy with carboplatin and etoposide or supportive care only. A multicentre randomised phase III trial. Eur J Cancer 1998;34: 1036-44.
Henshaw RC, Naji SA, Templeton AA. Comparison of medical abortion with surgical vacuum aspiration: women's preferences and acceptability of treatment. BMJ 1993;307: 714-7.
Kieler H, Hellberg D, Nilsson S, Waldenstr?m U, Axelsson O. Pregnancy outcome among non-participants in a trial on ultrasound screening. Ultrasound Obstet Gynecol 1998;11: 104-9.
King III SB, Barnhart HX, Kosinski AS, Weintraub WS, Lembo NJ, Petersen JY, et al. Angioplasty or surgery for multivessel coronary artery disease: comparison of eligible registry and randomized patients in the EAST trial and influence of treatment selection on outcomes. Am J Cardiol 1997;79: 1453-9.
Lidbrink E, Frisell J, Rosendahl I, Rutqvist LE. Nonattendance in the Stockholm mammography screening trial: relative mortality and reasons for nonattendance. Breast Cancer Res Treat 1995;35: 267-75.
Link MP, Goorin AM, Horowitz M, Meyer WH, Belasco J, Baker A, et al. Adjuvant chemotherapy of high-grade osteosarcoma of the extremity. Updated results of the multi-institutional osteosarcoma study. Clin Orthop Relat Res 1991;270: 8-14.
Liu WF, Harrington T. The need for delivery room intubation of thin meconium in the low risk newborn: a clinical trial. Am J Perinatol 1998;15: 675-82.
Mayo Asymptomatic Carotid Endarterectomy Study Group. Results of a randomized controlled trial of carotid endarterectomy for asymptomatic carotid stenosis. Mayo Clinic Proc 1992;67: 513-8.
Martinez-Amenos A, Ferre MLF, Vidal CM, Rocasalbas JA. Evaluation of two educative models in primary care hypertension programme. J Hum Hypertens 1990;4: 362-4.
McCaughey ES, Mulligan J, Voss LD, Betts PR. Randomised trial of growth hormone in short normal girls. Lancet 1998;351: 940-4.
McKay JR, Alterman AI, McLellan AT, Snider EC, O'Brian CP. Effect of random versus nonrandom assignment in a comparison of inpatient and day hospital rehabilitation for male alcoholics. J Consult Clin Psychol 1995;63: 70-8.
McKay JR, Alterman AI, McLellan AT, Boardman CR, Mulvaney FD, O'Brian CP. Random versus nonrandom assignment in the evaluation of treatment for cocaine abusers. J Consult Clin Psychol 1998;66: 697-701.
Moertel CG, Childs DS, O'Fallon JR, Holbrook MA, Schutt AJ, Reitemeier RJ. Combined 5-fluorouracil and radiation therapy as a surgical adjuvant for poor prognosis gastric carcinoma. J Clin Oncol 1984;2: 1249-54.
Mosekilde L, Beck-Nielsen H, S?rensen OH, Nielsen SP, Charles P, Vestergaard P, et al. Hormonal replacement therapy reduces forearm fracture incidence in recent postmenopausal women—results of the Danish osteoporosis prevention study. Maturitas 2000;36: 181-93.
Nagel HTC, Vanderbussche FPHA, Keirse MJNC, Oepkes D, Oosterwijk JC, Beverstock G, et al. Amniocentesis before 14 completed weeks as an alternative to transabdominal chorionic villus sampling: a controlled trial with infant follow-up. Prenat Diagn 1998;18: 465-75.
Nicolaides K, Brizot MdL, Patel F, Snijders R. Comparison of chorionic villus sampling and amniocentesis for fetal karyotyping at 10-13 weeks' gestation. Lancet 1994;344: 435-9.
Paradise JL, Bluestone CD, Bachman RZ, Colborn DK, Bernard BS, Taylor FH, et al. Efficacy of tonsillectomy for recurrent throat infection in severely affected children. Results of parallel randomized clinical trials. N Engl J Med 1984;310: 674-83.
Paradise JL, Bluestone CD, Rogers KD, Taylor FH, Colborn DK, Bachman RZ, et al. Efficacy of adenoidectomy for recurrent otitis media in children previously treated with tympanostomy-tube replacement. Results of parallel randomized and nonrandomized trials. JAMA 1990;263: 2066-73.
Playforth MJ, Smith GMR, Evans M, Pollock AV. Antimicrobial bowel preparation. Oral, parenteral, or both? Dis Colon Rectum 1988;31: 90-3.
Rigg JRA, Jamrozik K, Myles PS, Silbert B, Peyton P, Parsons RW, et al. Design of the multicenter Australian study of epidural anesthesia and analgesia in major surgery: the MASTER trial. Control Clin Trial 2000;21: 244-56.
Rosen MA, Roizen MF, Eger II EI, Glass RH, Martin M, Dandekar PV, et al. The effect of nitrous oxide on in vitro fertilization success rate. Anesthesiol 1987;67: 42-4.
Rovers MM, Straatman H, Ingels K, van der Wilt GJ, van den Broek P, Zielhuis GA. Generalizability of trial results based on randomized versus nonrandomized allocation of OME infants to ventilation tubes or watchful waiting. J Clin Epidemiol 2001;54: 789-94.
Schmoor C, Oleschewski M, Schumacher M. Randomized and non-randomized patients in clinical trials: experiences with comprehensive cohort studies. Stat Med 1996;15: 263-71.
Strandberg TE, Salomaa VV, Vanhanen HT, Naukkarin VA, Sarna AJ, Miettinen TA. Mortality in participants and non-participants of a multifactorial prevention study of cardiovascular diseases: a 28 year follow up of the Helsinki businessmen study. Br Heart J 1995;74: 449-54.
Sullivan MP, Fuller LM, Chen T, Fisher R, Fryer C, Gehan E, et al. Intergroup Hodgkin's disease in children study of stages I and II: a preliminary report. Cancer Treat Rep 1982;66: 937-47.
Urban P, Stauffer J-C, Bleed D, Khatchatrian N, Amann W, Bertel O, et al. A randomized evaluation of early revascularization to treat shock complicating acute myocardial infarction. Eur Heart J 1999;20: 1030-8.
Villamaria FJ, Baisden CE, Hillis A, Rajab MH, Rinaldi PA. Forced-air warming is no more effective than conventional methods for raising postoperative core temperature after cardiac surgery. J Cardiothorac Vasc Anesth 1997;11: 708-11.
Wetzner SM, Vincent ME, Robbins AH. Ceruletide-assisted cholecystography: a clinical assessment. Radiology 1979;131: 23-6.
Wikdahl AM, Granbom L, Stegmayr BG. CAPD bag changing with integrated disconnect system gives lower incidence of peritonitis than with UV-box system. Adv Perit Dial 1992;8: 276-80.
Williford WO, Krol WF, Buzby GP. Comparison of eligible randomized patients with two groups of ineligible patients: can the results of the VA total parenteral nutrition clinical trial be generalized? J Clin Epidemiol 1993;46: 1025-34.
Yamamoto H, Hughes RW, Schroeder KW, Viggiano TR, DiMagno EP. Treatment of benign esophageal stricture by Eder-Puestow or balloon dilators: a comparison between randomized and prospective nonrandomized trials. Mayo Clinic Proc 1992;67: 228-36.
Yersin B, Besson J, Duc-Mingot S, Burnand B. Screening and referral of alcoholic patients in a general hospital. A clinical trial. Eur Addict Res 1996;2: 94-101.
Young JDH, MacEwen CJ, Ogston SA. Congenital nasolacrimal duct obstruction in the second year of life: a multicentre trial of management. Eye 1996;10: 485-91.(Gunn Elisabeth Vist, senior researcher1,)
Correspondence to: G E Vist gev@nhsrc.no
Objective To systematically compare the outcomes of participants in randomised controlled trials (RCTs) with those in comparable non-participants who received the same or similar treatment.
Table 1 Summary of studies included in sensitivity analysis
Data sources Bibliographic databases, reference lists from eligible articles, medical journals, and study authors.
Review methods RCTs and cohort studies that evaluated the clinical outcomes of participants in RCTs and comparable non-participants who received the same or similar treatment.
Results Five RCTs (six comparisons) and 50 cohort studies (85 comparisons) provided data on 31 140 patients treated in RCTs and 20 380 comparable patients treated outside RCTs. In the five RCTs, in which patients were given the option of participating or not, the comparisons provided limited information because of small sample sizes (a total of 412 patients) and the nature of the questions considered. 73 dichotomous outcomes were compared, of which 59 reported no statistically significant differences. For patients treated within RCTs, 10 comparisons reported significantly better outcomes and four reported significantly worse outcomes. Significantly heterogeneity was found (I2 = 89%) among the comparisons of 73 dichotomous outcomes; none of our a priori explanatory factors helped explain this heterogeneity. The 18 comparisons of continuous outcomes showed no significant differences in heterogeneity (I2 = 0%). The overall pooled estimate for continuous outcomes of the effect of participating in an RCT was not significant (standardised mean difference 0.01, 95% confidence interval -0.10 to 0.12).
Conclusion No strong evidence was found of a harmful or beneficial effect of participating in RCTs compared with receiving the same or similar treatment outside such trials.
Properly conducted randomised controlled trials (RCTs) provide the strongest evidence of the effects of treatment.1 It is, however, controversial as to whether participants of such trials benefit directly or whether these studies are solely for the benefit of future patients. In addition, there is much scepticism about the applicability of the results to usual practice.2
Four reviews that considered whether it is beneficial or harmful to participate in RCTs drew varied conclusions.3-6 These reviews compared patients who were treated within trials with those treated outside the trials, regardless of differences between the clinical interventions or between the participants and non-participants. It is therefore uncertain whether the results reflect the effects of participating in an RCT (trial effects), differences in the clinical interventions (treatment effects), or differences between participants and non-participants. We determined whether the outcomes of participants in RCTs differed from those of comparable non-participants who received the same or similar treatment.
Methods
Our review was undertaken as a Cochrane methodology review (see Cochrane Library for fuller details of our methods and updated versions of the review).7
We included studies that compared participants in RCTs with comparable non-participants who received the same or similar treatment. We included observational studies and RCTs in which participation or the option of participation was randomly allocated.
Search strategies
We used seven strategies to identify relevant studies: consultation with experts; search of personal files; electronic searches of the Cochrane central register of controlled trials, Medline, Embase, the Cochrane methodology register, and PsycINFO; a review of reference lists from eligible articles; and a search of PubMed using the "related articles" feature and SciSearch. We also hand searched articles published in 2000 in five medical journals (BMJ, Annals of Internal Medicine, JAMA, Lancet, and New England Journal of Medicine) to identify RCTs with over 200 patients and at least 100 eligible non-participants. Studies' authors were contacted for data on the treatment and outcomes of eligible non-participants.
Assessment of study eligibility
Two reviewers independently assessed each article for eligibility. Disagreements were resolved by discussion. A third reviewer was consulted when consensus could not be reached.
Data ion
Two reviewers independently ed data from eligible studies. Each study was assessed for selection bias (differences between participants and non-participants), detection bias, and exclusion bias (losses to follow-up). On the basis of the combined risks of the three biases, we grouped each comparison into overall quality groups (randomised, controlled comparisons, partially controlled comparisons, and poorly controlled comparisons) for analysis. Missing data were sought from the investigators.
Analysis
We compared the experimental group of the RCT with their respective eligible non-participants who received the treatment and the control group and eligible non-participants who received the control treatment outside of the trial. For each comparison, we analysed the main outcome, and, when reported, we analysed mortality separately. For all of the included comparisons we used the main outcome as reported by the investigators. We analysed the dichotomous and continuous results separately. The results are reported as relative risks with 95% confidence intervals, using adjusted estimates when available. For a summary of the dichotomous results in one table, we calculated the associated standard error from the natural logarithm of the unadjusted relative risk. Heterogeneity was assessed by 2 test and the I2 statistic using RevMan version 4.2.8 For the unadjusted relative risk analysis, we used the Mantel-Haenszel test in RevMan. A fixed effect model was used to calculate summary statistics if no statistically significant (P < 0.10) heterogeneity was found among similar comparisons. For statistically heterogeneous results we described the variation in the estimates and key explanatory factors; where possible relating the explanatory factors to observed differences in estimates of the effects of participation. We constructed a funnel plot to explore the possibility of publication bias.
Fig 1 Selection process of eligible studies
Results
Overall, 55 studies, totalling 91 comparisons, met our inclusion criteria (fig 1). Forty one studies are still awaiting assessment, which currently cannot be included or excluded on the basis of the published information.
We identified five RCTs (six comparisons) in which patients were randomised according to whether they had the option to participate. These studies provided limited data because of their small sample sizes and the nature of the questions considered. Two studies randomised 82 patients to "n of 1 trials" compared with standard practice—that is, randomised, double blind, multiple crossover comparisons of an active drug with a placebo in a single patient.9 10 One study (60 patients) measured spontaneously self reported side effects in patients who had or had not been informed that they were in an RCT.11 One study (227 patients) reported satisfaction among patients randomised to an RCT compared with patients randomised to a patient preference trial in which they had a choice of treatment.12 Another study (43 patients) reported pain reduction among patients randomised to an RCT compared with those who were not invited to participate.13 None of these studies found significant differences in outcomes between patients treated in or outside RCTs.
Non-randomised cohort studies
We identified 50 cohort studies (85 comparisons) totalling 30 862 patients participating in RCTs compared with 20 246 patients treated outside RCTs.14-63 Seventy comparisons comprised dichotomous outcomes, of which 12 reported adjusted estimates, and 15 comparisons comprised continuous outcomes.
We found significant heterogeneity (I2 = 89.0%) among the results of comparisons with dichotomous main outcomes (fig 2); these results were therefore not pooled. Of these 73 comparisons, 59 reported no significant differences between outcomes for patients treated in RCTs and those receiving similar treatments outside RCTs; 10 reported significantly better outcomes for patients treated in RCTs, and four reported significantly worse outcomes for patients treated in RCTs.
Fig 2 Results of dichotomous main outcomes in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
Figure 3 shows the results of the 18 comparisons with continuous main outcomes. We found no significant heterogeneity (I2 = 0%). The pooled estimate found no differences in outcomes for patients treated in and outside RCTs (standardised mean difference 0.01, 95% confidence interval -0.10 to 0.12).
Fig 3 Results of continuous main outcomes in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
In 17 studies (32 comparisons) with data on mortality (fig 4), we found significant heterogeneity (I2 = 88.8%); the results were therefore not pooled. In 24 of the 32 comparisons we found no significant difference in mortality. Four comparisons reported a significant lower risk of mortality for patients treated in RCTs and four comparisons reported a significantly higher risk of mortality.
Fig 4 Comparison of mortality between participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
Separate subgroup analyses could not explain the observed heterogeneity by the different types of eligible non-participants, treatments, clinical specialities, or study quality (selection bias, detection bias, and exclusion bias). The table summarises the studies included in the sensitivity analysis.
The funnel plot of the dichotomous comparisons showed no asymmetry (fig 5), indicating a low risk of publication bias.
Fig 5 Funnel plot of dichotomous comparisons in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
Discussion
Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane methodology review). Issue 4. Oxford: Update Software, 2002.
Rothwell PM. External validity of randomised controlled trials: "To whom do the results of this trial apply?" Lancet 2005;365: 82-93.
Stiller CA. Centralised treatment, entry to trials and survival. Br J Cancer 1994;70: 352-62.
Braunholtz DA, Edwards SJL, Lilford RJ. Are randomized clinical trials good for us (in the short term)? Evidence for a "trial effect." J Clin Epidemiol 2001;54: 217-24.
The Emergency Care Research Institute 2002. Patients' reasons for participation in clinical trials and effect of trial participation on patient outcomes. www.ecri.org/Patient_Information/Patient_Reference_Guide/evidence.pdf (accessed Nov 2004).
Peppercorn JM, Weeks JC, Cook EFC, Joffe S. Comparison of outcomes in cancer patients treated within and outside clinical trials: conceptual framework and structured review. Lancet 2004;363: 263-70.
Vist GE, Hagen KB, Devereaux P, Bryant D, Kristoffersen DT, Oxman AD. Outcomes of patients who participate in randomised controlled trials compared to similar patients receiving similar interventions who do not participate (Cochrane methodology reviews). Issue 1. Oxford: Update Software, 2005.
Higgins JP, Thompson SG. Quantifying heterogeneity in meta-analysis. Stat Med 2002;21: 1539-58.
Mahon J, Laupacis A, Donner A, Wood T. Randomised study of n of 1 trials versus standard practice. BMJ 1996;312: 1069-74.
Mahon JL, Laupacis A, Hodder RV, McKim DA, Paterson NAM, Wood TE, et al. Theophylline for irreversible chronic airflow limitation. A randomized study comparing n of 1 trials to standard practice. Chest 1999;115: 38-48.
Dahan R, Caulin C, Figea L, Kanis JA, Caulin F, Segrestaa JM. Does informed consent influence therapeutic outcome? A clinical trial of the hypnotic activity of placebo in patients admitted to hospital. BMJ 1986;293: 363-4.
Cooper KG, Grant AM, Garratt AM. The impact of using a partially randomised patient preference design when evaluating alternative managements for heavy menstrual bleeding. Br J Obstet Gynaecol 1997;104: 1367-73.
Bergmann JF, Chassany O, Gandiol J, Deblois P, Kanis JA, Segrestaa JM, et al. A randomised clinical trial of the effect of informed consent on the analgesic activity of placebo and naproxen in cancer pain. Clin Trials Metaanal 1994;29: 41-7.
Balmukhanov SB, Beisebaev AA, Aitkoolova ZI, Mustaphin JS, Philippenko VI, Rismuhamedova RS, et al. Intramural and parametrial infusion of metronidazole in the radiotherapy of uterine cervix cancer: preliminary report. Int J Radiation Oncol Biol Phys 1989;16: 1061-3.
Baum E, Sather H, Nachman J, Seinfeld J, Kritvit W, Leikin S, et al. Relapse rates following cessation of chemotherapy during complete remission of acute lymphocytic leukemia: a report from childrens cancer study group. Med Pediatr Oncol 1979;7: 25-34.
Bedi N, Chilvers C, Churchill R, Dewey M, Duggan C, Fielding K, et al. Assessing effectiveness of treatment of depression in primary care: partially randomised preference trial. Br J Psychol 2000;177: 312-8.
Berglund G, Bolund C, Gustafsson UL, Sjoden PO. Is the wish to participate in a cancer rehabilitation program an indicator of the need? Comparisons of participants and non-participants in a randomized study. Psychooncology 1997;6: 35-46.
Bhattacharya S, Cameron IM, Mollison J, Parkin DE, Abramovich DR, Kitchener HC. Admission-discharge policies for hysteroscopic surgery: a randomised comparison of day case with in-patient admission. Eur J Obstet Gynecol Reprod Biol 1998;76: 81-4.
Bijker N, Peterse JL, Fentiman TS, Julien JP, Hart AAM, Avril A, et al. Effects of patient selection on the applicability of results from a randomised clinical trial (EORTC 10853) investigating breast-conserving treatment for DCIS. Breast conserving therapy for ductal carcinoma in situ. PhD Thesis, 2000.
Blichert-Toft M, Brincker H, Andersen JA, Andresen KW, Axelsson CK, Mouridsen HT, et al.. A Danish randomized trial comparing breast-conserving therapy with mastectomy in mammary carcinoma: preliminary results. Acta Oncologica 1988;27: 671-7.
Coronary artery surgery study (CASS): a randomised trial of coronary artery bypass surgery. Comparability of entry characteristics and survival in randomized patients and nonrandomized patients meeting randomization criteria. J Am Coll Cardiol 1984;3: 114-28.
Chauhan SP, Rutherford SE, Hess LW, Morrison JC. Prophylactic intrapartum amnioinfusion for patients with oligohydramnios: a prospective randomized study. J Reprod Med 1991;37: 817-20.
Chilvers C, Dewey M, Fielding K, Gretton V, Miller P, Palmer B, et al. Antidepressant drugs and generic counselling for treatment of major depression in primary care: randomised trial with patient preference arms. BMJ 2001;322: 772-5.
Clagett GP, Youkey JR, Bringham RA, Orecchia PM, Salander JM, Collins GJ, et al. Asymptomatic cervical bruit and abnormal ocular pneumoplethysmography: a prospective study comparing two approaches to management. Surgery 1984;96: 823-30.
Creutzig U, Ritter J, Zimmermann M, Schellong G. Does cranial irradiation reduce the risk of bone marrow relapse in acute myelogenous leukemia? Unexpected results of the childhood acute myelogenous leukemia study BFM-87. J Clin Oncol 1993;11: 279-86.
Davis S, Wright PW, Schulman SF, Hill LD, Pinkham RD, Johnson LP, et al. Participants in prospective, randomized clinical trials for resected non-small cell lung cancer have improved survival compared with nonparticipants in such trials. Cancer 1985;56: 1710-8.
Edsmyr F, Esposti PL, Johansson B, Strindberg B. Clinical experimental randomized study of 2.6-CIS-diphenylhexamethylcyclotetrasiloxane and estramurine-17-phosphate in the treatment of prostatic carcinoma. J Urol 1978;120: 705-7.
Feit F, Brooks MM, Sopko G, Keller NM, Rosen A, Krone R, et al. Long-term clinical outcome in the bypass angioplasty revascularization investigation registry: comparison with the randomized trial. Circulation 2000;101: 2795-802.
Forbes GM, Collins BJ. Nitrous oxide for colonoscopy: a randomized controlled study. Gastrointest Endosc 2000;51: 271-7.
Forssell C, Takolander R, Bergqvist D, Johansson A, Persson NH. Local versus general anaesthesia in carotid surgery. A prospective, randomised study. Eur J Vasc Surg 1989;3: 503-9.
Helsing M, Bergman B, Thaning L, Hero U, for the Joint Lung Cancer Study Group. Quality of life and survival in patients with advanced non-small cell lung cancer receiving supportive care plus chemotherapy with carboplatin and etoposide or supportive care only. A multicentre randomised phase III trial. Eur J Cancer 1998;34: 1036-44.
Henshaw RC, Naji SA, Templeton AA. Comparison of medical abortion with surgical vacuum aspiration: women's preferences and acceptability of treatment. BMJ 1993;307: 714-7.
Kieler H, Hellberg D, Nilsson S, Waldenstr?m U, Axelsson O. Pregnancy outcome among non-participants in a trial on ultrasound screening. Ultrasound Obstet Gynecol 1998;11: 104-9.
King III SB, Barnhart HX, Kosinski AS, Weintraub WS, Lembo NJ, Petersen JY, et al. Angioplasty or surgery for multivessel coronary artery disease: comparison of eligible registry and randomized patients in the EAST trial and influence of treatment selection on outcomes. Am J Cardiol 1997;79: 1453-9.
Lidbrink E, Frisell J, Rosendahl I, Rutqvist LE. Nonattendance in the Stockholm mammography screening trial: relative mortality and reasons for nonattendance. Breast Cancer Res Treat 1995;35: 267-75.
Link MP, Goorin AM, Horowitz M, Meyer WH, Belasco J, Baker A, et al. Adjuvant chemotherapy of high-grade osteosarcoma of the extremity. Updated results of the multi-institutional osteosarcoma study. Clin Orthop Relat Res 1991;270: 8-14.
Liu WF, Harrington T. The need for delivery room intubation of thin meconium in the low risk newborn: a clinical trial. Am J Perinatol 1998;15: 675-82.
Mayo Asymptomatic Carotid Endarterectomy Study Group. Results of a randomized controlled trial of carotid endarterectomy for asymptomatic carotid stenosis. Mayo Clinic Proc 1992;67: 513-8.
Martinez-Amenos A, Ferre MLF, Vidal CM, Rocasalbas JA. Evaluation of two educative models in primary care hypertension programme. J Hum Hypertens 1990;4: 362-4.
McCaughey ES, Mulligan J, Voss LD, Betts PR. Randomised trial of growth hormone in short normal girls. Lancet 1998;351: 940-4.
McKay JR, Alterman AI, McLellan AT, Snider EC, O'Brian CP. Effect of random versus nonrandom assignment in a comparison of inpatient and day hospital rehabilitation for male alcoholics. J Consult Clin Psychol 1995;63: 70-8.
McKay JR, Alterman AI, McLellan AT, Boardman CR, Mulvaney FD, O'Brian CP. Random versus nonrandom assignment in the evaluation of treatment for cocaine abusers. J Consult Clin Psychol 1998;66: 697-701.
Moertel CG, Childs DS, O'Fallon JR, Holbrook MA, Schutt AJ, Reitemeier RJ. Combined 5-fluorouracil and radiation therapy as a surgical adjuvant for poor prognosis gastric carcinoma. J Clin Oncol 1984;2: 1249-54.
Mosekilde L, Beck-Nielsen H, S?rensen OH, Nielsen SP, Charles P, Vestergaard P, et al. Hormonal replacement therapy reduces forearm fracture incidence in recent postmenopausal women—results of the Danish osteoporosis prevention study. Maturitas 2000;36: 181-93.
Nagel HTC, Vanderbussche FPHA, Keirse MJNC, Oepkes D, Oosterwijk JC, Beverstock G, et al. Amniocentesis before 14 completed weeks as an alternative to transabdominal chorionic villus sampling: a controlled trial with infant follow-up. Prenat Diagn 1998;18: 465-75.
Nicolaides K, Brizot MdL, Patel F, Snijders R. Comparison of chorionic villus sampling and amniocentesis for fetal karyotyping at 10-13 weeks' gestation. Lancet 1994;344: 435-9.
Paradise JL, Bluestone CD, Bachman RZ, Colborn DK, Bernard BS, Taylor FH, et al. Efficacy of tonsillectomy for recurrent throat infection in severely affected children. Results of parallel randomized clinical trials. N Engl J Med 1984;310: 674-83.
Paradise JL, Bluestone CD, Rogers KD, Taylor FH, Colborn DK, Bachman RZ, et al. Efficacy of adenoidectomy for recurrent otitis media in children previously treated with tympanostomy-tube replacement. Results of parallel randomized and nonrandomized trials. JAMA 1990;263: 2066-73.
Playforth MJ, Smith GMR, Evans M, Pollock AV. Antimicrobial bowel preparation. Oral, parenteral, or both? Dis Colon Rectum 1988;31: 90-3.
Rigg JRA, Jamrozik K, Myles PS, Silbert B, Peyton P, Parsons RW, et al. Design of the multicenter Australian study of epidural anesthesia and analgesia in major surgery: the MASTER trial. Control Clin Trial 2000;21: 244-56.
Rosen MA, Roizen MF, Eger II EI, Glass RH, Martin M, Dandekar PV, et al. The effect of nitrous oxide on in vitro fertilization success rate. Anesthesiol 1987;67: 42-4.
Rovers MM, Straatman H, Ingels K, van der Wilt GJ, van den Broek P, Zielhuis GA. Generalizability of trial results based on randomized versus nonrandomized allocation of OME infants to ventilation tubes or watchful waiting. J Clin Epidemiol 2001;54: 789-94.
Schmoor C, Oleschewski M, Schumacher M. Randomized and non-randomized patients in clinical trials: experiences with comprehensive cohort studies. Stat Med 1996;15: 263-71.
Strandberg TE, Salomaa VV, Vanhanen HT, Naukkarin VA, Sarna AJ, Miettinen TA. Mortality in participants and non-participants of a multifactorial prevention study of cardiovascular diseases: a 28 year follow up of the Helsinki businessmen study. Br Heart J 1995;74: 449-54.
Sullivan MP, Fuller LM, Chen T, Fisher R, Fryer C, Gehan E, et al. Intergroup Hodgkin's disease in children study of stages I and II: a preliminary report. Cancer Treat Rep 1982;66: 937-47.
Urban P, Stauffer J-C, Bleed D, Khatchatrian N, Amann W, Bertel O, et al. A randomized evaluation of early revascularization to treat shock complicating acute myocardial infarction. Eur Heart J 1999;20: 1030-8.
Villamaria FJ, Baisden CE, Hillis A, Rajab MH, Rinaldi PA. Forced-air warming is no more effective than conventional methods for raising postoperative core temperature after cardiac surgery. J Cardiothorac Vasc Anesth 1997;11: 708-11.
Wetzner SM, Vincent ME, Robbins AH. Ceruletide-assisted cholecystography: a clinical assessment. Radiology 1979;131: 23-6.
Wikdahl AM, Granbom L, Stegmayr BG. CAPD bag changing with integrated disconnect system gives lower incidence of peritonitis than with UV-box system. Adv Perit Dial 1992;8: 276-80.
Williford WO, Krol WF, Buzby GP. Comparison of eligible randomized patients with two groups of ineligible patients: can the results of the VA total parenteral nutrition clinical trial be generalized? J Clin Epidemiol 1993;46: 1025-34.
Yamamoto H, Hughes RW, Schroeder KW, Viggiano TR, DiMagno EP. Treatment of benign esophageal stricture by Eder-Puestow or balloon dilators: a comparison between randomized and prospective nonrandomized trials. Mayo Clinic Proc 1992;67: 228-36.
Yersin B, Besson J, Duc-Mingot S, Burnand B. Screening and referral of alcoholic patients in a general hospital. A clinical trial. Eur Addict Res 1996;2: 94-101.
Young JDH, MacEwen CJ, Ogston SA. Congenital nasolacrimal duct obstruction in the second year of life: a multicentre trial of management. Eye 1996;10: 485-91.(Gunn Elisabeth Vist, senior researcher1,)