What's the evidence that NICE guidance has been implemented? Results from a national evaluation using time series analysis, audit of patient
http://www.100md.com
《英国医生杂志》
1 Department of Health Sciences, University of York, York YO10 5DD, 2 York Health Economics Consortium, University of York, 3 Centre for Health Economics, University of York, 4 Bradford Royal Infirmary, Bradford, BD9 6RJ
Correspondence to: T A Sheldon tas5@york.ac.uk
Abstract
The National Institute for Clinical Excellence (NICE), established in 1999 by the Department of Health, aims to improve standards of care for patients and reduce inequalities in access to innovative treatments.1 NICE's technology appraisals result in guidance on the use of individual health technologies; around 50 for implementation by the NHS in England and Wales since 2000.2 Hitherto, guidance on treatments was uncoordinated. This nationally coordinated programme of guidance is probably unique and represents a policy embodiment of evidence based medicine that, it is hoped, will lead to the rapid and systematic uptake of evidence based medicine into routine practice. This paper presents the results of a national evaluation examining the pattern of implementation of NICE guidance by healthcare organisations. More details on NICE and a copy of the full report can be accessed via www.nice.org.uk.
Methods
Wisdom teeth extraction
National data indicate a sharp decline in the number of extractions between 1995 and 2001 (fig 1). Although these fell in the year in which the guidance had been published (in March 2000), we found no evidence of a change in the downward trend in extractions (-8.9 extractions per month, 95% confidence interval -57.5 to 39.6). Case note review showed that more than 90% of extractions were compliant with the guidance. Survey respondents indicated that compliance was high because the costs of implementation were low, involved a single specialty service, and had professional support and a strong evidence base.
Fig 1 Wisdom teeth extraction activity, 1992-2001
Hip prostheses
The guidance recommended the use of prostheses with a demonstrable replacement rate of 10% or less, at 10 years, or a minimum of three years, provided that the performance of the prostheses is consistent with the 10 year benchmark. The guidance did not specify which prostheses met the benchmark and the NHS Purchasing and Supply Agency was slow to issue this information (which was subsequently withdrawn).
Use of more than 50 different prostheses was documented in the Trent and Wales register over the period 1998-2002. Of the single prostheses, 69% (3671) met the 10 year benchmark and 81% (4327) met the three year benchmark, proportions that had been declining over recent years. There is some evidence that the decline in 10 year benchmarked prostheses may have stabilised slightly after guidance (1.7 monthly increase after guidance (95% confidence interval 0.4 to 3.1) compared with a long term monthly decrease of -1.8 prostheses per month (-0.75 to -2.80)). We observed similar results from the audit of patients' notes: 66% (325), 74% (727), and 75% (734) of heads, cups, and stems, respectively, meeting the benchmarks. Interviews and the survey indicated that surgeons thought that the guidance did not acknowledge the complexity of hip surgery, and some surgeons interviewed believed that cementless prostheses would ultimately prove to offer longer service.
Taxanes for breast and ovarian cancer
We obtained usable data on taxanes from 24 hospital pharmacies (including nine cancer centres). We found a significant increase in the use of docetaxel and paclitaxel of 1112 (95% confidence interval 530 to 2222, P < 0.001) patient months and 3.7 (1.1 to 7.8, P < 0.002) patient months, respectively (fig 2), but no evidence of a change in the growth rate in the use of gemcitabine (0.5% per month, -36.2% to 37.2%) or vinorelbine (-1.0 patient months, -3.3 to1.3). This is unlikely to be due to lack of statistical power.
Fig 2 Hospital use of paclitaxel (top) and docetaxel (bottom)
Case note review showed that, of the 707 patients identified as receiving taxanes for breast cancer, all but one were receiving this appropriately (for more advanced forms of the disease or in the context of randomised controlled trials). Interviewees acknowledged that the NICE guidance had made funding easier to obtain. The picture for ovarian cancer, however, was more varied. Of the 520 women with ovarian cancer for whom we have data, only 33% (166) were recorded as having been prescribed paclitaxel. However, oncologists interviewed believed that the guidance had overstated the effectiveness of taxanes in ovarian cancer. NICE subsequently amended its guidance.
Implantable cardioverter defibrillators
Although the number of implantable cardioverter defibrillators implanted has risen, we found no evidence of a significant change after NICE guidance had been published (fig 3). Given the small data set, the power is low to detect a change as significant, but visual inspection does not indicate any structural break. This may reflect the high costs of implantable cardioverter defibrillators, at around ￡20 000 ($36 000; 29) per device, competition for resources with other interventional procedures in cardiology, and scarcity of skills in electrophysiology.11
Fig 3 Total number of implantable cardioverter defibrillators implanted by quarter, 1995-2001
Hearing aids
The NICE guidance seems to have been received enthusiastically by audiology departments; all of those surveyed had undertaken an immediate audit of their service against the guidance requirements. However, review of case notes indicates that the range of analogue hearing aids offered does not seem to have been extended. Funding was described in the interviews as a major impediment to implementation. The guidance was issued at the same time as the Department of Health implemented a series of pilots of digital hearing aids, which cut across the guidance on analogue aids, which was subsequently withdrawn.
Laparoscopic surgery for primary inguinal hernia repair and colorectal cancer
Only 4% of primary inguinal hernia repairs in England and Wales were undertaken laparoscopically (contrary to NICE guidance), and this did not change after the guidance (0.3 monthly increase in hernia repairs, 95% confidence interval -5.48 to 6.08).
Although hospital episode statistics data for the 19 trusts that returned audit forms also showed 96% compliance, our audit of 545 repairs of primary unilateral hernias indicated only 65% compliance, indicating that coding of hospital episode statistics data may be unreliable. However, both national and audit data agreed that most laparoscopic procedures were concentrated in a few trusts, and that did not change over time.
Interviews showed that some local expert surgeons had the support of managers and commissioners to continue the use of laparoscopic surgery for primary repair. It was also claimed that patients often requested laparoscopic procedures.
The percentage of cases of colorectal cancer treated with laparoscopic surgery remained unchanged, at around 0.1% from 1998 to 2001.
Zanamivir for influenza
National prescribing data show little inappropriate prescribing of zanamivir in the absence of high levels of flu; prescriptions remained very low, at 499 in 2001, 190 in 2002, and 124 in 2003.
Orlistat for obesity
We found a significant increase in the average monthly prescribing of orlistat after the guidance had been published 22 per month (0.43, 95% confidence interval 15.9 to 27.8, P < 0.001; fig 4). Health authorities increased their use of orlistat, standardised by age, in the year after NICE guidance had been issued by about eight patient months per 1000 people aged 18-75 years. The variation in use, as measured by the coefficient of variation, fell from 0.4 in 2000-1 to 0.3 in 2001-2.
Fig 4 Use of orlistat in the community
In the 689 primary care patients for whom we have a record we found evidence that the drug was not being prescribed in accordance with guidance. Only in 12% of cases (n = 83) were there data showing compliance in the three key areas of age, body mass index, and weight loss. However, data recording was poor, and data were missing in one or more fields in 80% (551) of returns, particularly in respect of weight and weight loss. In those 308 cases where patients' weight loss prior to the visit had been recorded fully, the weight loss criterion was not met in 40% (123) of cases. Of patients with recorded weight loss data, only 41% (127) and 25% (40) still being prescribed orlistat at 3 months and 6 months, respectively, had reduced their weight as advised in the NICE guidance. The rise in prescribing of orlistat therefore does not necessarily imply a rise in appropriate prescribing.
Drugs for Alzheimer's disease
Total use grew logarithmically since February 1999 (fig 5). Once this had been taken into account, the growth rate at the time of publication of guidance (-1.4% per month; 95% confidence interval -4% to 1.2%) did not increase significantly. However, the data are also compatible with a view that an increase in trend occurred shortly before the guidance was formally issued. Variability between health authorities decreased; the coefficient of variation fell from 1.1 to 0.97.
Fig 5 Total use of Alzheimer's drugs in the community
Based on the audit data from 583 usable forms, compliance with the five recommendations in the guidance at first prescription varies between 52% and 85% for mental health organisations and 21% and 46% in primary care. Compliance with the recommendations at follow up is low, mainly because information regarding mini-mental state examination scores, follow ups, or the presence of relatives at assessment was not routinely recorded (a requirement of the guidance).
Discussion
Department of Health. Faster access to modern treatment. London: DoH, London, 1999.
Dillon A, Gibbs TG, Riley T, Sheldon TA. The National Institute for Clinical Excellence and coverage of Relenza by the NHS. In: Fox DM, Oxman AD, eds. Informing judgment: case studies of health policy and research in six countries. New York: Milbank Memorial Fund, 2001.
McDowall D, McCleary R, Meidinger EE, Hay RA. Interrupted time series analysis. Quantitative applications in the social sciences, volume 21. Thousand Oaks, CA: Sage, 1980.
Box G, Jenkins G. Time series analysis: forecasting and control. 2nd ed. San Francisco: Holden Day, 1984.
Park RE, Fink A, Brook RH, Chassin MR, Kahn KL, Merrick NJ, et al. Physician ratings of appropriate indications for three procedures: theoretical indications vs indications used in practice. Am J Public Health 1989;79: 445-7.
Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE. The reproducibility of a method to identify the overuse and underuse of procedures. N Engl J Med 1998;338: 1888-95.
Mays N, Pope C. Qualitative research in health care. London: BMJ Publications, 1999.
Silverman D. Doing qualitative research: a practical handbook. London: Sage, 2000.
Strauss A, Corbin J. Basics of qualitative research. 2nd ed. Thousand Oaks, CA: Sage, 1998.
Miles M, Huberman A. Qualitative data analysis. London: Sage, 1994.
Plummer CJ, McComb JM. An audit of the implications of implementing NICE guidance on the use of implantable cardioverter-defibrillators. Heart 2003;89: 787-8.
Mace S, Taylor D. Adherence to NICE guidance for the use of anticholinesterases for Alzheimer's disease. Pharmaceut J 2002;269: 680-1.
Bloor K, Freemantle N, Khadjesari Z, Maynard A. Impact of NICE guidance on laparoscopic surgery for inguinal hernias: analysis of interrupted time series. BMJ 2003;326: 578.
Brock D, Powell M, Hinings CR. Restructuring the professional organization: accounting, health care and law. London: Routledge, 1999.
Fitzgerald L, Ferlie E. Interlocking interactions, the diffusion of innovations in health care. Hum Relat 2002;55: 1429-49.
Rogers EM. Diffusion of innovations. 5th ed. New York: Free Press, 2003.
Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 2004;8(6).(Trevor A Sheldon, profess)
Correspondence to: T A Sheldon tas5@york.ac.uk
Abstract
The National Institute for Clinical Excellence (NICE), established in 1999 by the Department of Health, aims to improve standards of care for patients and reduce inequalities in access to innovative treatments.1 NICE's technology appraisals result in guidance on the use of individual health technologies; around 50 for implementation by the NHS in England and Wales since 2000.2 Hitherto, guidance on treatments was uncoordinated. This nationally coordinated programme of guidance is probably unique and represents a policy embodiment of evidence based medicine that, it is hoped, will lead to the rapid and systematic uptake of evidence based medicine into routine practice. This paper presents the results of a national evaluation examining the pattern of implementation of NICE guidance by healthcare organisations. More details on NICE and a copy of the full report can be accessed via www.nice.org.uk.
Methods
Wisdom teeth extraction
National data indicate a sharp decline in the number of extractions between 1995 and 2001 (fig 1). Although these fell in the year in which the guidance had been published (in March 2000), we found no evidence of a change in the downward trend in extractions (-8.9 extractions per month, 95% confidence interval -57.5 to 39.6). Case note review showed that more than 90% of extractions were compliant with the guidance. Survey respondents indicated that compliance was high because the costs of implementation were low, involved a single specialty service, and had professional support and a strong evidence base.
Fig 1 Wisdom teeth extraction activity, 1992-2001
Hip prostheses
The guidance recommended the use of prostheses with a demonstrable replacement rate of 10% or less, at 10 years, or a minimum of three years, provided that the performance of the prostheses is consistent with the 10 year benchmark. The guidance did not specify which prostheses met the benchmark and the NHS Purchasing and Supply Agency was slow to issue this information (which was subsequently withdrawn).
Use of more than 50 different prostheses was documented in the Trent and Wales register over the period 1998-2002. Of the single prostheses, 69% (3671) met the 10 year benchmark and 81% (4327) met the three year benchmark, proportions that had been declining over recent years. There is some evidence that the decline in 10 year benchmarked prostheses may have stabilised slightly after guidance (1.7 monthly increase after guidance (95% confidence interval 0.4 to 3.1) compared with a long term monthly decrease of -1.8 prostheses per month (-0.75 to -2.80)). We observed similar results from the audit of patients' notes: 66% (325), 74% (727), and 75% (734) of heads, cups, and stems, respectively, meeting the benchmarks. Interviews and the survey indicated that surgeons thought that the guidance did not acknowledge the complexity of hip surgery, and some surgeons interviewed believed that cementless prostheses would ultimately prove to offer longer service.
Taxanes for breast and ovarian cancer
We obtained usable data on taxanes from 24 hospital pharmacies (including nine cancer centres). We found a significant increase in the use of docetaxel and paclitaxel of 1112 (95% confidence interval 530 to 2222, P < 0.001) patient months and 3.7 (1.1 to 7.8, P < 0.002) patient months, respectively (fig 2), but no evidence of a change in the growth rate in the use of gemcitabine (0.5% per month, -36.2% to 37.2%) or vinorelbine (-1.0 patient months, -3.3 to1.3). This is unlikely to be due to lack of statistical power.
Fig 2 Hospital use of paclitaxel (top) and docetaxel (bottom)
Case note review showed that, of the 707 patients identified as receiving taxanes for breast cancer, all but one were receiving this appropriately (for more advanced forms of the disease or in the context of randomised controlled trials). Interviewees acknowledged that the NICE guidance had made funding easier to obtain. The picture for ovarian cancer, however, was more varied. Of the 520 women with ovarian cancer for whom we have data, only 33% (166) were recorded as having been prescribed paclitaxel. However, oncologists interviewed believed that the guidance had overstated the effectiveness of taxanes in ovarian cancer. NICE subsequently amended its guidance.
Implantable cardioverter defibrillators
Although the number of implantable cardioverter defibrillators implanted has risen, we found no evidence of a significant change after NICE guidance had been published (fig 3). Given the small data set, the power is low to detect a change as significant, but visual inspection does not indicate any structural break. This may reflect the high costs of implantable cardioverter defibrillators, at around ￡20 000 ($36 000; 29) per device, competition for resources with other interventional procedures in cardiology, and scarcity of skills in electrophysiology.11
Fig 3 Total number of implantable cardioverter defibrillators implanted by quarter, 1995-2001
Hearing aids
The NICE guidance seems to have been received enthusiastically by audiology departments; all of those surveyed had undertaken an immediate audit of their service against the guidance requirements. However, review of case notes indicates that the range of analogue hearing aids offered does not seem to have been extended. Funding was described in the interviews as a major impediment to implementation. The guidance was issued at the same time as the Department of Health implemented a series of pilots of digital hearing aids, which cut across the guidance on analogue aids, which was subsequently withdrawn.
Laparoscopic surgery for primary inguinal hernia repair and colorectal cancer
Only 4% of primary inguinal hernia repairs in England and Wales were undertaken laparoscopically (contrary to NICE guidance), and this did not change after the guidance (0.3 monthly increase in hernia repairs, 95% confidence interval -5.48 to 6.08).
Although hospital episode statistics data for the 19 trusts that returned audit forms also showed 96% compliance, our audit of 545 repairs of primary unilateral hernias indicated only 65% compliance, indicating that coding of hospital episode statistics data may be unreliable. However, both national and audit data agreed that most laparoscopic procedures were concentrated in a few trusts, and that did not change over time.
Interviews showed that some local expert surgeons had the support of managers and commissioners to continue the use of laparoscopic surgery for primary repair. It was also claimed that patients often requested laparoscopic procedures.
The percentage of cases of colorectal cancer treated with laparoscopic surgery remained unchanged, at around 0.1% from 1998 to 2001.
Zanamivir for influenza
National prescribing data show little inappropriate prescribing of zanamivir in the absence of high levels of flu; prescriptions remained very low, at 499 in 2001, 190 in 2002, and 124 in 2003.
Orlistat for obesity
We found a significant increase in the average monthly prescribing of orlistat after the guidance had been published 22 per month (0.43, 95% confidence interval 15.9 to 27.8, P < 0.001; fig 4). Health authorities increased their use of orlistat, standardised by age, in the year after NICE guidance had been issued by about eight patient months per 1000 people aged 18-75 years. The variation in use, as measured by the coefficient of variation, fell from 0.4 in 2000-1 to 0.3 in 2001-2.
Fig 4 Use of orlistat in the community
In the 689 primary care patients for whom we have a record we found evidence that the drug was not being prescribed in accordance with guidance. Only in 12% of cases (n = 83) were there data showing compliance in the three key areas of age, body mass index, and weight loss. However, data recording was poor, and data were missing in one or more fields in 80% (551) of returns, particularly in respect of weight and weight loss. In those 308 cases where patients' weight loss prior to the visit had been recorded fully, the weight loss criterion was not met in 40% (123) of cases. Of patients with recorded weight loss data, only 41% (127) and 25% (40) still being prescribed orlistat at 3 months and 6 months, respectively, had reduced their weight as advised in the NICE guidance. The rise in prescribing of orlistat therefore does not necessarily imply a rise in appropriate prescribing.
Drugs for Alzheimer's disease
Total use grew logarithmically since February 1999 (fig 5). Once this had been taken into account, the growth rate at the time of publication of guidance (-1.4% per month; 95% confidence interval -4% to 1.2%) did not increase significantly. However, the data are also compatible with a view that an increase in trend occurred shortly before the guidance was formally issued. Variability between health authorities decreased; the coefficient of variation fell from 1.1 to 0.97.
Fig 5 Total use of Alzheimer's drugs in the community
Based on the audit data from 583 usable forms, compliance with the five recommendations in the guidance at first prescription varies between 52% and 85% for mental health organisations and 21% and 46% in primary care. Compliance with the recommendations at follow up is low, mainly because information regarding mini-mental state examination scores, follow ups, or the presence of relatives at assessment was not routinely recorded (a requirement of the guidance).
Discussion
Department of Health. Faster access to modern treatment. London: DoH, London, 1999.
Dillon A, Gibbs TG, Riley T, Sheldon TA. The National Institute for Clinical Excellence and coverage of Relenza by the NHS. In: Fox DM, Oxman AD, eds. Informing judgment: case studies of health policy and research in six countries. New York: Milbank Memorial Fund, 2001.
McDowall D, McCleary R, Meidinger EE, Hay RA. Interrupted time series analysis. Quantitative applications in the social sciences, volume 21. Thousand Oaks, CA: Sage, 1980.
Box G, Jenkins G. Time series analysis: forecasting and control. 2nd ed. San Francisco: Holden Day, 1984.
Park RE, Fink A, Brook RH, Chassin MR, Kahn KL, Merrick NJ, et al. Physician ratings of appropriate indications for three procedures: theoretical indications vs indications used in practice. Am J Public Health 1989;79: 445-7.
Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE. The reproducibility of a method to identify the overuse and underuse of procedures. N Engl J Med 1998;338: 1888-95.
Mays N, Pope C. Qualitative research in health care. London: BMJ Publications, 1999.
Silverman D. Doing qualitative research: a practical handbook. London: Sage, 2000.
Strauss A, Corbin J. Basics of qualitative research. 2nd ed. Thousand Oaks, CA: Sage, 1998.
Miles M, Huberman A. Qualitative data analysis. London: Sage, 1994.
Plummer CJ, McComb JM. An audit of the implications of implementing NICE guidance on the use of implantable cardioverter-defibrillators. Heart 2003;89: 787-8.
Mace S, Taylor D. Adherence to NICE guidance for the use of anticholinesterases for Alzheimer's disease. Pharmaceut J 2002;269: 680-1.
Bloor K, Freemantle N, Khadjesari Z, Maynard A. Impact of NICE guidance on laparoscopic surgery for inguinal hernias: analysis of interrupted time series. BMJ 2003;326: 578.
Brock D, Powell M, Hinings CR. Restructuring the professional organization: accounting, health care and law. London: Routledge, 1999.
Fitzgerald L, Ferlie E. Interlocking interactions, the diffusion of innovations in health care. Hum Relat 2002;55: 1429-49.
Rogers EM. Diffusion of innovations. 5th ed. New York: Free Press, 2003.
Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 2004;8(6).(Trevor A Sheldon, profess)