Use of SPRAT for peer review of paediatricians in training
http://www.100md.com
《英国医生杂志》
1 Academic Unit of Child Health, Sheffield Children's Hospital, Sheffield S10 2HT, 2 Postgraduate Medical Education Department, Sheffield Children's Hospital, 3 Foundation for the Advancement of International Medical Education Research, (FAIMER), 3624 Market Street, Philadelphia, PA 19104, USA
Correspondence to: H A Davies h.davies@sheffield.ac.uk
Objective To determine whether a multisource feedback questionnaire, SPRAT (Sheffield peer review assessment tool), is a feasible and reliable assessment method to inform the record of in-training assessment for paediatric senior house officers and specialist registrars.
Design Trainees' clinical performance was evaluated using SPRAT sent to clinical colleagues of their choosing. Responses were analysed to determine variables that affected ratings and their measurement characteristics.
Setting Three tertiary hospitals and five secondary hospitals across a UK deanery.
Participants 112 paediatric senior house officers and middle grades.
Main outcome measures 95% confidence intervals for mean ratings; linear and hierarchical regression to explore potential biasing factors; time needed for the process per doctor.
Results 20 middle grades and 92 senior house officers were assessed using SPRAT to inform their record of in-training assessment; 921/1120 (82%) of their proposed raters completed a SPRAT form. As a group, specialist registrars (mean 5.22, SD 0.34) scored significantly higher (t = – 4.765) than did senior house officers (mean 4.81, SD 0.35) (P < 0.001). The grade of the doctor accounted for 7.6% of the variation in the mean ratings. The hierarchical regression showed that only 3.4% of the variation in the means could be additionally attributed to three main factors (occupation of rater, length of working relationship, and environment in which the relationship took place) when the doctor's grade was controlled for (significant F change < 0.001). 93 (83%) of the doctors in this study would have needed only four raters to achieve a reliable score if the intent was to determine if they were satisfactory. The mean time taken to complete the questionnaire by a rater was six minutes. Just over an hour of administrative time is needed for each doctor.
Conclusions SPRAT seems to be a valid way of assessing large numbers of doctors to support quality assurance procedures for training programmes. The feedback from SPRAT can also be used to inform personal development planning and focus quality improvements.
Lipner RS, Blank LL, Leas BF, Fortna GS. The value of patient and peer ratings in recertification. Acad Med 2002;77(10 suppl): S64-6.
Archer JC, Davies HA. Clinical management. Where medicine meets management: on reflection. Health Serv J 2004;114(5903): 26-7.
Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA 1993;269: 1655-60.
Hall W, Violato C, Lewkonia R, Lockyer J, Fidler H, Toews J, et al. Assessment of physician performance in Alberta: the physician achievement review. CMAJ 1999;161: 52-7.
Thomas PA, Gebo KA, Hellmann DB. A pilot study of peer review in residency training. J Gen Intern Med 1999;14: 551-4.
Evans R, Elwyn G, Edwards A. Review of instruments for peer assessment of physicians. BMJ 2004;328: 1240.
Archer JC, Davies HA. Sheffield peer review assessment tool for consultants (SPRAT): screening for poorly performing doctors. Bern, Switzerland: Association of Medical Education of Europe, 2003.
General Medical Council. Good medical practice. London: GMC, 2001.
Royal College of Paediatrics and Child Health. Good medical practice in paediatrics and child health: duties and responsibilities of paediatricians. London: Royal College of Paediatrics and Child Health, 2002.
Violato C, Marini A, Towes J, Fidler H. Feasibility and psychometric properties of using peers, consulting physicians, co-workers and patients to assess physicians. Acad Med 1997;72(suppl 1): S82-4.
Violato C, Lockyer J, Fidler H. Multisource feedback: a method of assessing surgical practice. BMJ 2003;326: 546-8.
Davis JD. Comparison of faculty, peer, self, and nurse assessment of obstetrics and gynecology residents. Obstet Gynecol 2002;99: 647-51.
Cronbach LJ, Shavelson RJ. My current thoughts on coefficient alpha and successor procedures. Educ Psychol Measurem 2004;64: 391-418.
Holmboe ES. Faculty and the observation of trainees' clinical skills: problems and opportunities. Acad Med 2004;79: 16-22.(Julian C Archer, clinical research fello)
Correspondence to: H A Davies h.davies@sheffield.ac.uk
Objective To determine whether a multisource feedback questionnaire, SPRAT (Sheffield peer review assessment tool), is a feasible and reliable assessment method to inform the record of in-training assessment for paediatric senior house officers and specialist registrars.
Design Trainees' clinical performance was evaluated using SPRAT sent to clinical colleagues of their choosing. Responses were analysed to determine variables that affected ratings and their measurement characteristics.
Setting Three tertiary hospitals and five secondary hospitals across a UK deanery.
Participants 112 paediatric senior house officers and middle grades.
Main outcome measures 95% confidence intervals for mean ratings; linear and hierarchical regression to explore potential biasing factors; time needed for the process per doctor.
Results 20 middle grades and 92 senior house officers were assessed using SPRAT to inform their record of in-training assessment; 921/1120 (82%) of their proposed raters completed a SPRAT form. As a group, specialist registrars (mean 5.22, SD 0.34) scored significantly higher (t = – 4.765) than did senior house officers (mean 4.81, SD 0.35) (P < 0.001). The grade of the doctor accounted for 7.6% of the variation in the mean ratings. The hierarchical regression showed that only 3.4% of the variation in the means could be additionally attributed to three main factors (occupation of rater, length of working relationship, and environment in which the relationship took place) when the doctor's grade was controlled for (significant F change < 0.001). 93 (83%) of the doctors in this study would have needed only four raters to achieve a reliable score if the intent was to determine if they were satisfactory. The mean time taken to complete the questionnaire by a rater was six minutes. Just over an hour of administrative time is needed for each doctor.
Conclusions SPRAT seems to be a valid way of assessing large numbers of doctors to support quality assurance procedures for training programmes. The feedback from SPRAT can also be used to inform personal development planning and focus quality improvements.
Lipner RS, Blank LL, Leas BF, Fortna GS. The value of patient and peer ratings in recertification. Acad Med 2002;77(10 suppl): S64-6.
Archer JC, Davies HA. Clinical management. Where medicine meets management: on reflection. Health Serv J 2004;114(5903): 26-7.
Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA 1993;269: 1655-60.
Hall W, Violato C, Lewkonia R, Lockyer J, Fidler H, Toews J, et al. Assessment of physician performance in Alberta: the physician achievement review. CMAJ 1999;161: 52-7.
Thomas PA, Gebo KA, Hellmann DB. A pilot study of peer review in residency training. J Gen Intern Med 1999;14: 551-4.
Evans R, Elwyn G, Edwards A. Review of instruments for peer assessment of physicians. BMJ 2004;328: 1240.
Archer JC, Davies HA. Sheffield peer review assessment tool for consultants (SPRAT): screening for poorly performing doctors. Bern, Switzerland: Association of Medical Education of Europe, 2003.
General Medical Council. Good medical practice. London: GMC, 2001.
Royal College of Paediatrics and Child Health. Good medical practice in paediatrics and child health: duties and responsibilities of paediatricians. London: Royal College of Paediatrics and Child Health, 2002.
Violato C, Marini A, Towes J, Fidler H. Feasibility and psychometric properties of using peers, consulting physicians, co-workers and patients to assess physicians. Acad Med 1997;72(suppl 1): S82-4.
Violato C, Lockyer J, Fidler H. Multisource feedback: a method of assessing surgical practice. BMJ 2003;326: 546-8.
Davis JD. Comparison of faculty, peer, self, and nurse assessment of obstetrics and gynecology residents. Obstet Gynecol 2002;99: 647-51.
Cronbach LJ, Shavelson RJ. My current thoughts on coefficient alpha and successor procedures. Educ Psychol Measurem 2004;64: 391-418.
Holmboe ES. Faculty and the observation of trainees' clinical skills: problems and opportunities. Acad Med 2004;79: 16-22.(Julian C Archer, clinical research fello)