On bias and transparency in the development of influential recommendations
http://www.100md.com
《加拿大医疗协会学报》
Clinical practice guidelines and reimbursement recommendations such as those of the Canadian Diabetes Association (CDA) and the Canadian Expert Drug Advisory Committee (CEDAC) about insulin glargine1,2 have a potentially great effect on clinical practice. Minimizing bias during their development is therefore at least as important as it is during clinical trials. A conflict-of-interest guideline is only one method of minimizing bias. In this article I will review the potential for bias that exists in the Common Drug Review (CDR) process (Box 1) and contrast this with my understanding of the development of the CDA guidelines.
The CDR includes conflict-of-interest guidelines.3 Annually, all CEDAC members complete a detailed form describing all interactions with any party that might be considered conflicts of interest during the past 5 years. These are shared with the CEDAC chair; at each meeting, it is decided whether a potential conflict exists that should exclude a member from participating in the review of a drug. No CEDAC member owns personal stock in any drug company, and none has received pharmaceutical company support of any kind related to a drug being considered by CEDAC.
The many potential sources of bias in the clinical literature almost always result in overestimation of benefit and underestimation of harm. These include preferentially reporting positive studies,4 more frequently reporting the outcomes measured in a trial that improve with therapy than those that do not,5 changing the prespecified primary outcome after a trial is analyzed5 and paying less attention to harm than to efficacy.6 A voluntary registry of clinical trials is a first step toward decreasing publication bias,7 but only mandatory registration of all trials before approval by any research ethics board will eliminate this bias. Other sources of bias will be avoided only when the protocols of clinical trials are registered at the time the studies begin, when all the results of all clinical trials are made public at the time a drug is licensed,8 and when the side effects of a drug are more rigorously assessed in clinical trials. Progress in these areas has been painfully slow, but I am delighted that the Pharmaceutical Research and Manufacturers of America has endorsed the routine submission of trial protocols.9 I will be even more delighted when it happens, and all protocols become public.
The CDR uses standard methodology to perform a detailed independent systematic review of the available literature about the benefits, harms and cost-effectiveness of each drug under consideration for reimbursement by publicly funded drug-benefit plans.10 These reviews, which attempt to summarize the literature in as unbiased a manner as possible, are shared with the manufacturer for comment before the CEDAC meeting. They are not made public, however, in part because they often contain summaries of information from manufacturers that is confidential. CDA guidelines describe a method of grading their recommendations, but they provide little detail about the methodology used to review the literature and do not provide a critical appraisal of the various randomized trials of insulin glargine.1
Interpreting the literature is not the same as summarizing it: interpretation inevitably incorporates an individual's values and perspectives. CEDAC's mandate is to make reimbursement recommendations from the perspective of the health care system, based not only upon a drug's effectiveness but also its cost-effectiveness.10 Its 11 members are physicians, pharmacists and nurses, chosen to represent a variety of clinical backgrounds, all with the ability to critically and sensibly interpret clinical findings and cost-effectiveness data. Moreover, each CDR review involves at least one external reviewer who is a content expert. In the case of insulin glargine, the clinical expert was an endocrinologist who had declared receiving honoraria from the manufacturer of insulin glargine that totaled less than $4000.
The CDA has complained that because no member of CEDAC is an endocrinologist, the committee underestimated the impact of insulin glargine upon patient care.11,12 I would argue that the CEDAC reviewers, both internal and external, met the most important criterion needed to fulfill their mandate: to be able to review the available literature competently and objectively. In addition, the committee's clinicians had sufficient clinical experience with diabetes to assess the benefits of insulin glargine. Indeed, CEDAC's documented reasons for recommendation acknowledged that insulin glargine decreased the frequency of hypoglycemia in several clinical trials. The reason CEDAC recommended against reimbursement was that the relatively small improvement in hypoglycemia was not felt to justify the drug's more than 3-fold price relative to NPH (neutral protamine Hagedorn) insulin.2
Finally, it is important to consider whether the structure within which experts work could lead to bias or the perception of bias. CEDAC members are appointed by federal, provincial or territorial deputy ministers of health and are paid an honorarium by the CDR; public-drug-plan managers are allowed to observe CEDAC meetings; and CEDAC has no formal interaction with members of the public. In addition, CEDAC reports to the Board of the Canadian Coordinating Office of Health Technology Assessment (CCOHTA), which is made up entirely of representatives of the federal/provincial/ territorial Ministries of Health. No wonder a recent external assessment of the CDR found that members of advocacy groups representing people with various diseases called for greater public involvement in the CDR process.13 On the other hand, the CDA guidelines were sponsored by pharmaceutical and diagnostic companies, the methods of reviewing and summarizing the literature were not fully described, and the potential conflicts of interest of authors are unknown. Neither the guidelines nor the CDA letter of protest to the Canadian Ministers of Health about CEDAC's recommendation1,12 acknowledged the 3-fold price differential associated with insulin glargine compared with NPH insulin. No wonder those who pay for drugs are concerned that groups such as the CDA sometimes preferentially emphasize the evidence that supports their position and minimize the evidence that does not.
Although I believe that the CDR process of reviewing and interpreting the available literature is as unbiased as possible, transparency would be enhanced if the CDR's reviews were made public. Most of the concerns about reporting biases that have been highlighted previously are beyond CEDAC's ability to correct. In terms of transparency, CEDAC's conflict-of-interest guidelines are publicly available.3 The CDR Web site will soon be modified to indicate that CEDAC members are paid for their work, and it will include a summary of each CEDAC member's declared potential conflicts of interest. Finally, CCOHTA is considering methods of greater public involvement in the CDR process; this cannot occur soon enough.
@ See related articles pages 333 and 337
Footnotes
Acknowledgement: The views expressed in this article are those of the author and do not represent the views of other members of the Common Drug Review, the Canadian Expert Drug Advisory Committee, the Institute for Clinical Evaluative Sciences or the Ontario Ministry of Health and Long-Term Care.
Competing interests: The author receives fees and travel expenses from the Common Drug Review for his work as chair of the Canadian Expert Drug Advisory Committee and from Novartis for working as a member of Data Safety Monitoring Boards.
REFERENCES
Canadian Diabetes Association Clinical Practice Guidelines Expert Committee. Canadian Diabetes Association 2003 clinical practice guidelines for the prevention and management of diabetes in Canada. Can J Diabetes 2003:27(Suppl 2):S1-152. Available: www.diabetes.ca/cpg2003 (accessed 2005 Dec 12).
Canadian Coordinating Office for Health Technology Assessment (CCOHTA). CEDAC final recommendation on reconsideration and reasons for recommendation — insulin glargine. Available: www.ccohta.ca/CDR/cdr_pdf/cdr
CCOHTA. Conflict of interest guidelines for the Common Drug Review. Available:
www.ccohta.ca/CDR/cdr_pdf/CDR_Conflict_Interest_Guidelines.pdf (accessed 2005 Dec 21).
Easterbrook PJ, Berlin JA, Gopalan A, et al. Publication bias in clinical research. Lancet 1991;337:867-72.
Chan AW, Hrobjartsson A, Haahr MT, et al. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published studies. JAMA 2004;291:2457-65.
Chou R, Helfand M. Challenges in systematic reviews that assess treatment harms. Ann Intern Med 2005;142:1090-9.
De Angelis CD, Drazen JM, Frizelle FA, et al. Is this clinical trial fully registered? A statement from the International Committee of Medical Editors. CMAJ 2005;172(13):1700-2. Available: www.cmaj.ca/cgi/content/full/172/13/1700 (accessed 2005 Dec 20).
Report of the Expert Advisory Panel on the Safety of Cox-2 Selective Non-steroidal Anti-inflammatory Drugs (NSAIDs). Available: www.hc-sc.gc.ca/dhp-mps/alt_formats/hpfb-dgpsa/pdf/prodpharma/sap_report_gcs_rapport_cox2_e.pdf (accessed 2005 Dec 11).
Loew CJ. Conflicts of interest and independent data analysis in industry-funded studies [comment on JAMA 2005;294:110-1.]. JAMA 2005;294:2575, author reply 2576-7.
Canadian Coordinating Office for Health Technology Assessment. Common Drug Review — process. Available: www.ccohta.ca/entry_e.html (accessed 2005 Dec 11).
Canadian Diabetes Association. Canadian Diabetes Association concerned over recommendation to deny reimbursement for a new insulin diabetes treatment option [news release]. Available: www.diabetes.ca/Section_Main/NewsReleases.asp?ID=124 (accessed 2005 Dec 9).
Howlett MC. Letter to the Honourable Ujjal Dosanjh, Minister of Health. Available: www.diabetes.ca/files/Federal-Glargiine-letter-final.pdf (accessed 2005 Dec 11).
EKOS Research Associates Inc. Common Drug Review (CDR) — evaluation. Available: www.ccohta.ca/CDR/cdr_evaluation_e.cfm (accessed 2005 Dec 11).(Andreas Laupacis)
The CDR includes conflict-of-interest guidelines.3 Annually, all CEDAC members complete a detailed form describing all interactions with any party that might be considered conflicts of interest during the past 5 years. These are shared with the CEDAC chair; at each meeting, it is decided whether a potential conflict exists that should exclude a member from participating in the review of a drug. No CEDAC member owns personal stock in any drug company, and none has received pharmaceutical company support of any kind related to a drug being considered by CEDAC.
The many potential sources of bias in the clinical literature almost always result in overestimation of benefit and underestimation of harm. These include preferentially reporting positive studies,4 more frequently reporting the outcomes measured in a trial that improve with therapy than those that do not,5 changing the prespecified primary outcome after a trial is analyzed5 and paying less attention to harm than to efficacy.6 A voluntary registry of clinical trials is a first step toward decreasing publication bias,7 but only mandatory registration of all trials before approval by any research ethics board will eliminate this bias. Other sources of bias will be avoided only when the protocols of clinical trials are registered at the time the studies begin, when all the results of all clinical trials are made public at the time a drug is licensed,8 and when the side effects of a drug are more rigorously assessed in clinical trials. Progress in these areas has been painfully slow, but I am delighted that the Pharmaceutical Research and Manufacturers of America has endorsed the routine submission of trial protocols.9 I will be even more delighted when it happens, and all protocols become public.
The CDR uses standard methodology to perform a detailed independent systematic review of the available literature about the benefits, harms and cost-effectiveness of each drug under consideration for reimbursement by publicly funded drug-benefit plans.10 These reviews, which attempt to summarize the literature in as unbiased a manner as possible, are shared with the manufacturer for comment before the CEDAC meeting. They are not made public, however, in part because they often contain summaries of information from manufacturers that is confidential. CDA guidelines describe a method of grading their recommendations, but they provide little detail about the methodology used to review the literature and do not provide a critical appraisal of the various randomized trials of insulin glargine.1
Interpreting the literature is not the same as summarizing it: interpretation inevitably incorporates an individual's values and perspectives. CEDAC's mandate is to make reimbursement recommendations from the perspective of the health care system, based not only upon a drug's effectiveness but also its cost-effectiveness.10 Its 11 members are physicians, pharmacists and nurses, chosen to represent a variety of clinical backgrounds, all with the ability to critically and sensibly interpret clinical findings and cost-effectiveness data. Moreover, each CDR review involves at least one external reviewer who is a content expert. In the case of insulin glargine, the clinical expert was an endocrinologist who had declared receiving honoraria from the manufacturer of insulin glargine that totaled less than $4000.
The CDA has complained that because no member of CEDAC is an endocrinologist, the committee underestimated the impact of insulin glargine upon patient care.11,12 I would argue that the CEDAC reviewers, both internal and external, met the most important criterion needed to fulfill their mandate: to be able to review the available literature competently and objectively. In addition, the committee's clinicians had sufficient clinical experience with diabetes to assess the benefits of insulin glargine. Indeed, CEDAC's documented reasons for recommendation acknowledged that insulin glargine decreased the frequency of hypoglycemia in several clinical trials. The reason CEDAC recommended against reimbursement was that the relatively small improvement in hypoglycemia was not felt to justify the drug's more than 3-fold price relative to NPH (neutral protamine Hagedorn) insulin.2
Finally, it is important to consider whether the structure within which experts work could lead to bias or the perception of bias. CEDAC members are appointed by federal, provincial or territorial deputy ministers of health and are paid an honorarium by the CDR; public-drug-plan managers are allowed to observe CEDAC meetings; and CEDAC has no formal interaction with members of the public. In addition, CEDAC reports to the Board of the Canadian Coordinating Office of Health Technology Assessment (CCOHTA), which is made up entirely of representatives of the federal/provincial/ territorial Ministries of Health. No wonder a recent external assessment of the CDR found that members of advocacy groups representing people with various diseases called for greater public involvement in the CDR process.13 On the other hand, the CDA guidelines were sponsored by pharmaceutical and diagnostic companies, the methods of reviewing and summarizing the literature were not fully described, and the potential conflicts of interest of authors are unknown. Neither the guidelines nor the CDA letter of protest to the Canadian Ministers of Health about CEDAC's recommendation1,12 acknowledged the 3-fold price differential associated with insulin glargine compared with NPH insulin. No wonder those who pay for drugs are concerned that groups such as the CDA sometimes preferentially emphasize the evidence that supports their position and minimize the evidence that does not.
Although I believe that the CDR process of reviewing and interpreting the available literature is as unbiased as possible, transparency would be enhanced if the CDR's reviews were made public. Most of the concerns about reporting biases that have been highlighted previously are beyond CEDAC's ability to correct. In terms of transparency, CEDAC's conflict-of-interest guidelines are publicly available.3 The CDR Web site will soon be modified to indicate that CEDAC members are paid for their work, and it will include a summary of each CEDAC member's declared potential conflicts of interest. Finally, CCOHTA is considering methods of greater public involvement in the CDR process; this cannot occur soon enough.
@ See related articles pages 333 and 337
Footnotes
Acknowledgement: The views expressed in this article are those of the author and do not represent the views of other members of the Common Drug Review, the Canadian Expert Drug Advisory Committee, the Institute for Clinical Evaluative Sciences or the Ontario Ministry of Health and Long-Term Care.
Competing interests: The author receives fees and travel expenses from the Common Drug Review for his work as chair of the Canadian Expert Drug Advisory Committee and from Novartis for working as a member of Data Safety Monitoring Boards.
REFERENCES
Canadian Diabetes Association Clinical Practice Guidelines Expert Committee. Canadian Diabetes Association 2003 clinical practice guidelines for the prevention and management of diabetes in Canada. Can J Diabetes 2003:27(Suppl 2):S1-152. Available: www.diabetes.ca/cpg2003 (accessed 2005 Dec 12).
Canadian Coordinating Office for Health Technology Assessment (CCOHTA). CEDAC final recommendation on reconsideration and reasons for recommendation — insulin glargine. Available: www.ccohta.ca/CDR/cdr_pdf/cdr
CCOHTA. Conflict of interest guidelines for the Common Drug Review. Available:
www.ccohta.ca/CDR/cdr_pdf/CDR_Conflict_Interest_Guidelines.pdf (accessed 2005 Dec 21).
Easterbrook PJ, Berlin JA, Gopalan A, et al. Publication bias in clinical research. Lancet 1991;337:867-72.
Chan AW, Hrobjartsson A, Haahr MT, et al. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published studies. JAMA 2004;291:2457-65.
Chou R, Helfand M. Challenges in systematic reviews that assess treatment harms. Ann Intern Med 2005;142:1090-9.
De Angelis CD, Drazen JM, Frizelle FA, et al. Is this clinical trial fully registered? A statement from the International Committee of Medical Editors. CMAJ 2005;172(13):1700-2. Available: www.cmaj.ca/cgi/content/full/172/13/1700 (accessed 2005 Dec 20).
Report of the Expert Advisory Panel on the Safety of Cox-2 Selective Non-steroidal Anti-inflammatory Drugs (NSAIDs). Available: www.hc-sc.gc.ca/dhp-mps/alt_formats/hpfb-dgpsa/pdf/prodpharma/sap_report_gcs_rapport_cox2_e.pdf (accessed 2005 Dec 11).
Loew CJ. Conflicts of interest and independent data analysis in industry-funded studies [comment on JAMA 2005;294:110-1.]. JAMA 2005;294:2575, author reply 2576-7.
Canadian Coordinating Office for Health Technology Assessment. Common Drug Review — process. Available: www.ccohta.ca/entry_e.html (accessed 2005 Dec 11).
Canadian Diabetes Association. Canadian Diabetes Association concerned over recommendation to deny reimbursement for a new insulin diabetes treatment option [news release]. Available: www.diabetes.ca/Section_Main/NewsReleases.asp?ID=124 (accessed 2005 Dec 9).
Howlett MC. Letter to the Honourable Ujjal Dosanjh, Minister of Health. Available: www.diabetes.ca/files/Federal-Glargiine-letter-final.pdf (accessed 2005 Dec 11).
EKOS Research Associates Inc. Common Drug Review (CDR) — evaluation. Available: www.ccohta.ca/CDR/cdr_evaluation_e.cfm (accessed 2005 Dec 11).(Andreas Laupacis)