The MS Symptom and Impact Diary (MSSID): psychometric evaluation of a new instrument to measure the day to day impact of multiple sclerosis
http://www.100md.com
《神经病学神经外科学杂志》
1 Health Care Practice R&D Unit, University of Salford, Allerton Building, Frederick Road, Salford M6 4WT, UK
2 United Leeds Teaching Hospital’s Trust, Beckett Street, Leeds LS9 7TF, UK
3 Nuffield Institute for Health, University of Leeds, 71–75 Clarendon Road, Leeds LS2 9PL, UK
ABSTRACT
Objectives: This study aimed to develop further a diary originally devised to measure the impact of multiple sclerosis (MS) as part of a cost utility study of beta interferon, and to evaluate its reliability, validity, and responsiveness in an outpatient sample of people with MS.
Methods: The original diary was further developed using qualitative and quantitative methods to ensure that it addressed the views of people with MS. The psychometric properties of the MS Symptom and Impact Diary (MSSID) were evaluated in a sample of 77 people who completed the MSSID daily for 12 weeks. Internal and test–retest reliability, discriminant and convergent validity, and responsiveness were assessed using traditional psychometric methods.
Results: The MSSID formed three, internally consistent scales that measured mobility, fatigue, and the overall impact of MS. The test–retest reliability of the mobility scale was adequate for individual comparisons (ICC>0.90) and the fatigue and overall impact scales were adequate for group comparisons (ICC>0.70). The MSSID was able to distinguish between clinical groups depending on clinical course, indoor ambulation status, and relapse status. It demonstrated associations with other single point instruments in the expected direction. Compared with single point instruments, its responsiveness was similar or better, especially in detecting short term improvements in functioning.
Conclusions: The MSSID may provide a useful complement to currently available instruments to measure the outcomes of MS within clinical trials. Further research is needed to explore its feasibility in the context of a randomised controlled trial and its utility for clinicians.
Keywords: multiple sclerosis; psychometrics; diary
Abbreviations: BI, Barthel Index; MS, multiple sclerosis; LHS, London Handicap Scale; Leeds MS quality of life scale; MSIS-29, MS Symptom and Impact Scale-29; MSSID, MS Symptom and Impact Diary
The importance of measuring outcomes from the perspective of people with multiple sclerosis (MS) has recently become more widely recognised.1,2 People with MS have different concerns to clinicians regarding the most important areas in which MS affects their lives; clinicians place more importance on the physical aspects of the condition, while patients are more concerned with mental health and vitality.3 The relationship between clinical assessments of disability and those of the patient have also typically been low.4,5 Moreover, disease modifying treatments for MS, such as beta interferon, may produce benefits that are significant to the patient but are not measured by clinical assessments.6
The content of outcome measures needs to reflect the issues that people with MS consider to be important.7 It is only recently that disease specific instruments to measure the outcomes of MS have been developed using a patient centred approach. These include instruments such as the Leeds MS quality of life scale (LMSQoL)8 and the MS Symptom and Impact Scale-29 (MSIS-29)9,10 both of which have been shown to be reliable, valid, and responsive to change.
Such instruments represent a significant contribution to the measurement of outcomes from the perspective of people with MS. However, they are not equipped to measure a central feature of MS as a disease that has a significant effect on the lives of people with MS, namely the day to day variability of symptoms and their impact.11,12 The variable and unpredictable nature of MS makes it difficult for people with MS to plan their lives or be spontaneous, and results in increased levels of psychological distress,13,14 but is rarely measured in clinical trials or in clinical practice. The ambiguous and transient nature of many MS symptoms can also make it difficult for people with MS and for clinicians to identify when a relapse has occurred and when recovery begins.15,16
There is a need for new ways of assessing the outcomes of MS that take account of its variable nature. Increased importance should be placed on the use of symptom diaries in trials to accurately document the onset and resolution of relapses and symptoms.15 Diaries have been frequently used as outcome measures in randomised controlled trials of chronic diseases.17–19 They can provide a more comprehensive view of people’s health problems and enable the investigation of short and long term changes in health patterns over time.18–20 They have also been used to measure the quality of life of people with MS more comprehensively within a study of the cost utility of beta interferon, but as yet, the psychometric properties of this diary have not been tested.21 This paper reports on a study that modified and extended this diary,21 with reference to the views of people with MS, and assessed its validity, reliability, and responsiveness in an outpatient sample of people with MS using traditional psychometric methods.23,24
METHODS
Development of the MSSID
The MSSID was developed further in three phases. Firstly, two focus groups (n = 8 and n = 5) and ten in depth interviews with a purposively selected sample of people with MS, recruited from a community based population, explored the impact of MS and the relevance of diaries. Their views were used to modify the original diary.21 Secondly, the modified diary was pre-tested with nine people with MS using "think aloud" interviews,25 in which respondents gave a verbal report of their thought processes as they completed the diary to identify difficulties with the phrasing or response options of the items. Thirdly, the refined diary was piloted with 14 people with MS recruited from outpatient clinics, who completed the diary for 14 consecutive days. The final version of the MSSID (see appendix 1), in the form of an A5 booklet, comprised eight items. Two items (Q1 and Q2) asked about the experience and impact of 14 MS related problems on daily activities, four items (Q4–Q6) asked about the overall impact of MS, one item (Q7) compared health today with health yesterday, and the final item (Q8) left space for respondents to note down anything else that they considered to be important. This version of the MSSID formed the focus for psychometric assessment.
Participants
The psychometric properties of the MSSID were assessed in a sample of 77 people with MS recruited from outpatient clinics in two centres, between September and October 2000. People with MS attending the clinic were eligible if they were diagnosed with MS, aware of their diagnosis, and aged between 18 and 65 years. Those unable to complete a questionnaire owing to cognitive impairment were excluded. Those who had been diagnosed with MS for less than 1 year and those who were judged by the clinician to be experiencing difficulties dealing with their diagnosis were also excluded, to avoid the possibility of causing undue distress. The study was approved by the relevant local research ethics committees, and all participants gave written informed consent before taking part.
Instruments
At recruitment, the mobility status of all participants was rated according to whether they moved around the clinic unaided or used a walking aid. Participants were asked to complete three consecutive, 4 week diaries. At baseline and at the end of the 12 week study, participants completed a postal questionnaire. This included the LMSQoL8 as a measure of quality of life; the MSQoL-5427 as a measure of health status (the sexual functioning items were excluded as they have been found to be poorly completed26 and did not reflect the content of the diary); the postal Barthel Index28 as a measure of disability/functioning; and the London Handicap Scale (LHS)29 and the Reintegration to Normal Living Index (RNLI)30 as measures of handicap or participation. At the end of each 4 week diary period, participants completed a questionnaire asking about changes to symptoms and MS overall, medication, and relapses using questions from the cost utility study.21
Data quality
Data quality was assessed by examining response rates, missing items, and score distributions.
Reliability
Both internal and test-retest reliability were assessed. The internal structure of the MSSID was examined using principal components analysis with varimax rotation on three randomly selected days, one from each 4 week diary period. Two sets of analyses were performed, one for the symptom items (Q2) and one for the impact items (Q3–Q6). Factors were entered into the rotation if they had an eigenvalue of 1 or more, and loadings of 0.4 and above were used to determine which items belonged to which factor.31 The item content of the resulting factors were independently reviewed by a consultant neurologist (HF) to ensure they made clinical sense. Factor scores for each day were calculated by summing the items in each factor and then dividing by the total number of items completed within the factor that day. Higher scores indicated better functioning.
The internal reliability of the resulting factors was assessed using Cronbach’s alpha32 and item total correlations on three randomly selected days, one from each month. Reliability was deemed adequate if Cronbach’s alphas were >0.824 and item total correlations >0.2.23
The test-retest reliability of the resulting factors was assessed in two ways using the intraclass correlation coefficient (ICC).33 Firstly, the ICC was calculated between median week 1 and week 4 factor scores in stable respondents—that is, those whose symptoms did not change and who did not have a relapse during the first 4 weeks (n = 26). Secondly, the ICC was calculated between factor scores on the first 2 days of each 4 week diary period when the respondent indicated that their health had not changed (n = 61). ICCs between 0.70 and 0.89 were deemed adequate for group comparisons, while ICCs of 0.9 and above were deemed adequate for individual comparisons.24
Validity
Both discriminant and convergent construct validity were assessed using appropriate summary indicators of the longitudinal diary data.34 To provide a summary indicator of the impact of symptoms on daily activities (factors 1 and 2) and the impact of MS overall (factor 3), median factor scores were used. The median was chosen rather than the mean as it was expected that the median would be less influenced by missing data, and it was not expected that an individual’s scores over time would be normally distributed.
Previous research has demonstrated that people with progressive MS experience greater levels of impairment and disability than other disease course groups,35 and that people who have had a relapse experience worse health status than those who have not.21 Therefore, it was hypothesised that there would be statistically significant differences in median factor scores between those with relapsing remitting or benign MS v those with progressive MS, between those who walked unaided v those who used a walking aid and between those who experienced a relapse during the study v those who did not. In each case it was expected that for patients with progressive MS, those who used a walking aid and those who had a relapse would have worse (lower) median factor scores.
The convergent validity of the MSSID was examined by exploring correlations between the median factor scores during the last 4 week diary period and single point instruments collected at the end of the study. Correlations <0.4 were considered small, between 0.41 and 0.7 moderate, and 0.71 and above high.9
Responsiveness
Effect sizes of the change in median MSSID scores, LMSQoL, and MSQol-54 scores between week 1 and week 4 were calculated separately for those who indicated that their MS had got better (n = 14) and those who indicated that their MS had got worse (n = 20) in the first 4 weeks of the study.36 Effect sizes of 0.2 were considered to be small, 0.5 moderate, and 0.8 or more large.37 It was expected that effect sizes for the MSSID scores and MSQol-54 would be positive for those who got better and negative for those who got worse. The reverse pattern was expected for the LMSQoL because higher scores indicate worse functioning.
Statistical analyses
The differences in continuous variables (for example, MSSID scores) were assessed using Mann-Whitney U tests. Differences in categorical variables were assessed using the 2 test. Relationships between MSSID factor scores and single point instruments were assessed using Spearman’s rank correlations. All analyses were carried out in SPSS for Windows (version 9).
RESULTS
Changes made to the original diary
A number of changes were made to the original diary following the developmental stages. Seven symptom items were added to reflect issues that were raised in focus groups and interviews (for example "difficulties thinking clearly or forgetting things" and "feeling frustrated"). Some items were reworded to make them more interpretable to respondents (for example "altered sensations" was changed to "altered or loss of sensation"). The number of response options of some items was increased to better reflect the variation in symptom impact and health perceptions (for example, the response options for Q6 were reworded and increased from three to six). Finally, an open ended question (Q8) was added to allow respondents to record any additional issues or events that happened.
Sample characteristics
Sixty three people (82%) returned all three diaries; these were classed as respondents and used in subsequent analyses. Eighty one percent were women, the average age was 46 years, and they had been diagnosed with MS for an average of 11 years (table 1). Seventy three percent were currently not in paid work and 33% had experienced a relapse during the study. There were no statistically significant differences in age, gender, length of time since diagnosis, or clinical course between respondents recruited from either centre, thus the two samples were analysed together.
Table 1 Clinical and demographic characteristics of the sample
Data quality
Across the entire 12 weeks, 7% of MSSID items were missing, ranging from 5.8% to 7.6% across items. Three quarters of respondents completed at least 93% of the items and all respondents completed at least 50% of the items. Three symptom impact items (Q2) "weakness or heaviness in your legs", "problems with your bladder and bowel", and "feeling frustrated", had floor effects, with 20% or more person days rated as "a lot".
Internal reliability
The factor analysis of symptom impact items (Q2) revealed three factors that explained between 62% and 68% of the total variance. However, one factor contained only two items ("blurred, double, or shaky vision" and "coughing or choking when eating and drinking") and performed poorly in the internal reliability analyses, thus it was not retained. The remaining two symptom impact factors measured "mobility" and "fatigue". The factor analysis of the overall impact items (Q3–Q6) formed one factor that explained between 66% and 79% of the total variance and measured the overall impact of MS. For the three retained factors, Cronbach’s alpha exceeded or approached 0.8 and item total correlations exceeded 0.2 (table 2).
Table 2 Internal and test–retest reliability analysis of the MSSID
Test–retest reliability
For both weekly factor median scores and daily factor score comparisons, the ICC for fatigue and overall impact exceeded 0.7 and for mobility, exceeded 0.9 (table 2).
Discriminant validity
There were statistically significant differences in the overall impact scores between all the clinical groups in the expected direction (table 3). For mobility, there were statistically significant differences between all clinical groups in the expected direction with the exception of the relapse/no relapse comparison. For fatigue there were statistically significant differences between all clinical groups in the expected direction with the exception of the relapsing remitting/benign v progressive MS comparison. There were no statistically significant differences in the median factor scores between those recruited from either centre (data available on request).
Table 3 Discriminant validity analysis of the MSSID
Convergent validity
Table 4 shows the Spearman’s rank correlation coefficients between median factor scores during the last four weeks and the single point instruments collected at the end of the study. Mobility showed moderate correlations with the MSQol-54 physical and social functioning scales and with the RNLI. Fatigue showed moderate correlations with the MSQol-54 energy and emotional wellbeing scales and the RNLI. Overall impact of MS showed moderate correlations with the RNLI, the LMSQoL, MSQol-54 social function, and overall quality of life. The factor scores were all highly related to each other and were more highly related to each other than to the single point instruments. Fatigue and mobility were more highly related to each other than to overall impact of MS.
Table 4 Spearman’s rank correlations between median MSSID factor scores for weeks 9–12 and single point instruments collected at week 12
Responsiveness
Effect sizes for those who got better were moderate for MSSID scores and small for the MSQol-54 and LMSQol scores with one exception (emotional role limitations) (table 5). Effect sizes for those who got worse were small for the MSSID, MSQoL-54, and LMSQoL scores. The effect sizes were in the expected direction for MSSID and LMSQoL scores, but did not show a consistent pattern for MSQoL-54 scores.
Table 5 Effect sizes for median MSSID scores, MSQol-54, and LMSQoL scores for those who got better or worse in the first 4 weeks of the study
DISCUSSION
This study has provided initial support for the reliability, validity, and responsiveness of the (MSSID). Response rates were similar but the rates of missing items were higher compared with other studies using the MSQol-54, the LMSQoL, and the MSIS-29.8,9,27 However, response rates and rates of missing data were similar to other diary studies,18,22 providing initial evidence for the diary’s feasibility and acceptability. None of the items had ceiling effects and only three items had floor effects. These three items, "weakness or heaviness in your legs", "problems with your bladder and bowel", and "feeling frustrated" were retained because the developmental work indicated that they are both common and important problems for people with MS. Further work will evaluate whether excluding these items has an impact on the psychometric properties of the MSSID.
The majority of items in the MSSID focus on the physical symptoms of MS and their impact on daily activities, which is similar to the item content of the MSIS-29.9 However, like the MSIS-29, it also includes important items that address the emotional impact of the condition. The MSSID consists of three factors that measure mobility, the cognitive and emotional aspects of fatigue, and the overall impact of MS. These factors are internally reliable and their test–retest reliability was adequate for group comparisons for weekly mobility, fatigue, and overall impact scores, and individual comparisons for daily mobility and fatigue scores. The MSSID was able to discriminate between people with progressive and relapsing remitting MS, between those who used a walking aid and those who did not, and between those who had recently experienced a relapse and those who had not. It demonstrated expected moderate associations with appropriate scales on the MSQoL-54, the LMSQoL, the Barthel Index, the LHS, and the RNLI. Although there was some overlap in the constructs measured by the factor scores, they assess the impact of mobility, fatigue, and MS as a whole on daily activities.
Compared with the MSQoL-54 and the LMSQoL, the responsiveness of the median weekly MSSID scores was similar or better. The MSSID was better able to detect short term improvements in functioning than worsening, which may be attributable to the floor effects in the three items. It was also more responsive to short term improvements in functioning than the MSQoL-54 and the LMSQoL. This is in contrast to a previous study that found an asthma diary to have lower responsiveness compared with an asthma questionnaire. The better short term responsiveness of the MSSID may be because it was explicitly designed to detect short term changes in functioning, whereas the MSQoL-54 and the LMSQoL were not. The MSSID thus provides a valuable complement to currently existing instruments in its measurement of short term changes and day to day fluctuations in functioning.
The sample size used in the study was small compared with those used to develop the LMSQoL, the MSIS-29, and the MSQoL-54.8,9,26 However, it was expected to be sufficient to provide accurate estimates of the psychometric properties of the MSSID because sample sizes as low as 20 for reliability estimates and 40 for validity estimates did not alter the psychometric interpretation of the MSIS-29 compared with a sample size of 713.38 The sample was recruited from outpatient clinics, and a third of respondents experienced a relapse during the study, reflecting previous research that people with MS attending outpatient clinics are more likely to experience relapses or have greater disability for their age than community based populations.39 Therefore, further testing of the psychometric properties of the MSSID in a community based sample of people with MS would be valuable.
This study suggests that the MSSID could be used in clinical trials of both disease modifying drugs and interventions to manage MS symptoms. The MSSID may be a useful tool to monitor the occurrence of relapses and measure their impact in a psychometrically robust way. The MSSID is likely to require more resources to collect and manage the data and further research is required to assess the feasibility of its use in the context of a randomised controlled trial. However, as symptom diaries are sometimes used in current trials of MS therapies15 but are rarely psychometrically evaluated, the MSSID represents a significant advance in this respect.
The MSSID may also be useful in routine clinical practice in providing the patient with a prompt to inform clinicians about new symptoms or a worsening of existing symptoms, thus reducing the uncertainty around the occurrence of a relapse.16 It can also provide a graphical display of changes in the impact of symptoms over time and may be useful to clinicians in monitoring responses to treatment. Further work is needed to explore the most appropriate and feasible method of presenting MSSID data to clinicians and to examine clinicians’ and patients’ views of the utility of this data.
CONCLUSIONS
This study provided initial evidence that the MSSID is reliable, valid and responsive to change. It may provide a useful complement to currently available instruments to measure the outcomes of MS within clinical trials or clinical practice. Further research is needed to explore its feasibility and utility to clinicians in randomised controlled clinical trials and within routine clinical practice.
ACKNOWLEDGEMENTS
The authors would like to thank A Jacoby for permission to modify the diary from the cost utility study and her comments on the study design. We would also like to thank C Young, M Boggild, and M Johnson for their comments on the study design and for access to and recruitment of people with MS to the study. D Greenwood and S Walters provided statistical advice. Thanks also go to all the people who gave their time to participate in the research. J Greenhalgh was an MRC Training Fellow in Health Services Research 1997–2001.
REFERENCES
Rothwell PM. Quality of life in multiple sclerosis. J Neurol Neurosurg Psychiatry 1998;65:433.
Devinsky O. Outcome research in neurology: incorporating health related quality of life. Ann Neurol 1995;37:141–2.
Rothwell PM, McDowell Z, Wong CK, et al. Doctors and patients don’t agree: Cross sectional study of patients’ and doctors’ perceptions and assessments of disability in multiple sclerosis. BMJ 1997;314:1580–3.
Hoogervorst E, Eikelenboom M, Uitdehaag BMJ, et al. One year changes in disability in multiple sclerosis: neurological examination compared with patient self report. J Neurol Neurosurg Psychiatry 2003;74:439–42.
Cheng EM, Hays RD, Myers LW, et al. Factors related to agreement between self-reported and conventional Expanded Disability Status Scale (EDSS) scores. Mult Scler 2001;7:405–10.
Nortvedt MW, Riise T, Myhr KM, et al. Type I interferons and the quality of life of multiple sclerosis patients. Results from a clinical trial on interferon alfa-2a. Mult Scler 1999;5:317–22.
Greenhalgh J, Long A, Brettle AJ, et al. Reviewing and selecting outcome measures for use in routine practice. J Eval Clin Pract 1998;4:339–50.
Ford HL, Gerry E, Tennant A, et al. Developing a disease-specific quality of life measure for people with multiple sclerosis. Clinl Rehabil 2001;15:247–58.
Hobart JC, Lamping DL, Fitzpatrick R. The Multiple Sclerosis Impact Scale (MSIS-29) a new patient based outcome measure. Brain 2001;124:962–73.
Riazi A, Hobart JC, Lamping DL, et al. Multiple Sclerosis Impact Scale (MSIS-29): reliability and validity in hospital based samples. J Neurol Neurosurg Psychiatry 2002;73:701–4.
Miller C. The lived experience of multiple sclerosis. J Neurosci Nurs 1997;29:294–304.
Koopman W, Schweitzer A. The journey to multiple sclerosis: a qualitative study. J Neurosci Nurs 1999;31:17–26.
Mullins LL, Cote MP, Fuemmeler BF, et al. Illness intrusiveness, uncertainty and distress in individuals with multiple sclerosis. Rehabil Psychol 2001;46:139–53.
Kroencke D, Denny D, Lynch SG. Depression during exacerbations in multiple sclerosis: the importance of uncertainty. Mult Scler 2001;7:237–42.
Lui C, Blumhardt LD. Assessing relapses in treatment trials of relapsing and remitting multiple sclerosis: can we do better? Mult Scler 1999;5:22–8.
Baker L. Sense making in multiple sclerosis: the information needs of people with MS during an acute exacerbation. Qual Health Res 1998;8:106–20.
Sandha GS, Hunt RH, Van Zanten SJOV. A systematic overview of the use of diary cards, quality-of-life questionnaires, and psychometric tests in treatment trials of Helicobacter pylori-positive and -negative non-ulcer dyspepsia. Scand J Gastroenterol 1999;34:244–9.
Fayers P. MRC quality of life studies using a daily diary card—practical lessons learned from cancer trials. Qual Life Res 1995;4:343–52.
Juniper EF, O’Byrne PM, Ferrie PJ, et al. Measuring asthma control. Clinic questionnaire or daily diary? Am J Respir Crit Care Med 2000;162:1330–4.
Verbrugge LM, Balaban DJ. Patterns of change in disability and well being. Med Care 1989;27 (Suppl 3) :S128–47.
Parkin D, Jacoby A, McNamee P, et al. Treatment of multiple sclerosis with interferon beta: An appraisal of cost-effectiveness and quality of life. J Neurol Neurosurg Psychiatry 2000;68:144–9.
Verbrugge LM. Health diaries. Med Care 1980;18:73–95.
Streiner D, Norman G. Health measurement scales: a practical guide to their development and use, 2nd ed. Oxford: Oxford University Press, 1996.
Nunally J, Bernstein IH. Psychometric theory, 3rd ed. New York: McGraw-Hill, 1994.
Ericsson KA, Simon HA. Verbal reports as data. Psychol Rev 1980;87:215–51.
Vickrey BG, Hays RD, Harooni R, et al. A health-related quality of life measure for multiple sclerosis. Qual Life Res 1995;4:187–206.
Freeman JA, Hobart JC, Thompson AJ. Does adding MS-specific items to a generic measure (the SF-36) improve measurement? Neurology 2001;57:68–74.
Mahoney FI, Barthel DW. Functional evaluation: the Barthel Index. Md Med 1965;14:61–5.
Harwood RH, Rogers A, Dickinson E, et al. Measuring handicap: the London Handicap Scale, a new outcome measure for chronic disease. Qual Health Care 1994;3:11–16.
Wood-Dauphinee S, Opzoomer MA, Williams JI, et al. Assessment of global function: the Reintegration to Normal Living Index. Arch Phys Med Rehabil 1987;69:583–90.
Child D. Essentials of factor analysis, 2nd ed. London: Cassell, 1990.
Cronbach IJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297–334.
Armitage P, Berry G, Matthews JNS. Statistical methods in medical research, 4th ed. Oxford: Blackwell, 2002.
Matthews JNS, Altman DG, Campbell MJ, et al. Analysis of serial measurements in medical research. BMJ 1990;30:230–5.
Ford HL, Gerry E, Johnson MH, et al. Health status and quality of life of people with multiple sclerosis. Disabil Rehabil 2001;23:516–21.
Kazis L, Anderson J, Meenan R. Effect sizes for interpreting changes in health status. Med Care 1989;27 (Suppl 3):S178–89.
Cohen J. Statistical power analysis for the behavioural sciences. New York: Academic Press, 1977.
Hobart JC. What sample size for reliability and validity? Qual Life Res 2002;11:636 (Abstract).
Nelson LM, Franklin GM, Hanman RF, et al. Referral bias in multiple sclerosis research. J Clin Epidemiol 1988;41:187–92.(J Greenhalgh1, H Ford2, A)
2 United Leeds Teaching Hospital’s Trust, Beckett Street, Leeds LS9 7TF, UK
3 Nuffield Institute for Health, University of Leeds, 71–75 Clarendon Road, Leeds LS2 9PL, UK
ABSTRACT
Objectives: This study aimed to develop further a diary originally devised to measure the impact of multiple sclerosis (MS) as part of a cost utility study of beta interferon, and to evaluate its reliability, validity, and responsiveness in an outpatient sample of people with MS.
Methods: The original diary was further developed using qualitative and quantitative methods to ensure that it addressed the views of people with MS. The psychometric properties of the MS Symptom and Impact Diary (MSSID) were evaluated in a sample of 77 people who completed the MSSID daily for 12 weeks. Internal and test–retest reliability, discriminant and convergent validity, and responsiveness were assessed using traditional psychometric methods.
Results: The MSSID formed three, internally consistent scales that measured mobility, fatigue, and the overall impact of MS. The test–retest reliability of the mobility scale was adequate for individual comparisons (ICC>0.90) and the fatigue and overall impact scales were adequate for group comparisons (ICC>0.70). The MSSID was able to distinguish between clinical groups depending on clinical course, indoor ambulation status, and relapse status. It demonstrated associations with other single point instruments in the expected direction. Compared with single point instruments, its responsiveness was similar or better, especially in detecting short term improvements in functioning.
Conclusions: The MSSID may provide a useful complement to currently available instruments to measure the outcomes of MS within clinical trials. Further research is needed to explore its feasibility in the context of a randomised controlled trial and its utility for clinicians.
Keywords: multiple sclerosis; psychometrics; diary
Abbreviations: BI, Barthel Index; MS, multiple sclerosis; LHS, London Handicap Scale; Leeds MS quality of life scale; MSIS-29, MS Symptom and Impact Scale-29; MSSID, MS Symptom and Impact Diary
The importance of measuring outcomes from the perspective of people with multiple sclerosis (MS) has recently become more widely recognised.1,2 People with MS have different concerns to clinicians regarding the most important areas in which MS affects their lives; clinicians place more importance on the physical aspects of the condition, while patients are more concerned with mental health and vitality.3 The relationship between clinical assessments of disability and those of the patient have also typically been low.4,5 Moreover, disease modifying treatments for MS, such as beta interferon, may produce benefits that are significant to the patient but are not measured by clinical assessments.6
The content of outcome measures needs to reflect the issues that people with MS consider to be important.7 It is only recently that disease specific instruments to measure the outcomes of MS have been developed using a patient centred approach. These include instruments such as the Leeds MS quality of life scale (LMSQoL)8 and the MS Symptom and Impact Scale-29 (MSIS-29)9,10 both of which have been shown to be reliable, valid, and responsive to change.
Such instruments represent a significant contribution to the measurement of outcomes from the perspective of people with MS. However, they are not equipped to measure a central feature of MS as a disease that has a significant effect on the lives of people with MS, namely the day to day variability of symptoms and their impact.11,12 The variable and unpredictable nature of MS makes it difficult for people with MS to plan their lives or be spontaneous, and results in increased levels of psychological distress,13,14 but is rarely measured in clinical trials or in clinical practice. The ambiguous and transient nature of many MS symptoms can also make it difficult for people with MS and for clinicians to identify when a relapse has occurred and when recovery begins.15,16
There is a need for new ways of assessing the outcomes of MS that take account of its variable nature. Increased importance should be placed on the use of symptom diaries in trials to accurately document the onset and resolution of relapses and symptoms.15 Diaries have been frequently used as outcome measures in randomised controlled trials of chronic diseases.17–19 They can provide a more comprehensive view of people’s health problems and enable the investigation of short and long term changes in health patterns over time.18–20 They have also been used to measure the quality of life of people with MS more comprehensively within a study of the cost utility of beta interferon, but as yet, the psychometric properties of this diary have not been tested.21 This paper reports on a study that modified and extended this diary,21 with reference to the views of people with MS, and assessed its validity, reliability, and responsiveness in an outpatient sample of people with MS using traditional psychometric methods.23,24
METHODS
Development of the MSSID
The MSSID was developed further in three phases. Firstly, two focus groups (n = 8 and n = 5) and ten in depth interviews with a purposively selected sample of people with MS, recruited from a community based population, explored the impact of MS and the relevance of diaries. Their views were used to modify the original diary.21 Secondly, the modified diary was pre-tested with nine people with MS using "think aloud" interviews,25 in which respondents gave a verbal report of their thought processes as they completed the diary to identify difficulties with the phrasing or response options of the items. Thirdly, the refined diary was piloted with 14 people with MS recruited from outpatient clinics, who completed the diary for 14 consecutive days. The final version of the MSSID (see appendix 1), in the form of an A5 booklet, comprised eight items. Two items (Q1 and Q2) asked about the experience and impact of 14 MS related problems on daily activities, four items (Q4–Q6) asked about the overall impact of MS, one item (Q7) compared health today with health yesterday, and the final item (Q8) left space for respondents to note down anything else that they considered to be important. This version of the MSSID formed the focus for psychometric assessment.
Participants
The psychometric properties of the MSSID were assessed in a sample of 77 people with MS recruited from outpatient clinics in two centres, between September and October 2000. People with MS attending the clinic were eligible if they were diagnosed with MS, aware of their diagnosis, and aged between 18 and 65 years. Those unable to complete a questionnaire owing to cognitive impairment were excluded. Those who had been diagnosed with MS for less than 1 year and those who were judged by the clinician to be experiencing difficulties dealing with their diagnosis were also excluded, to avoid the possibility of causing undue distress. The study was approved by the relevant local research ethics committees, and all participants gave written informed consent before taking part.
Instruments
At recruitment, the mobility status of all participants was rated according to whether they moved around the clinic unaided or used a walking aid. Participants were asked to complete three consecutive, 4 week diaries. At baseline and at the end of the 12 week study, participants completed a postal questionnaire. This included the LMSQoL8 as a measure of quality of life; the MSQoL-5427 as a measure of health status (the sexual functioning items were excluded as they have been found to be poorly completed26 and did not reflect the content of the diary); the postal Barthel Index28 as a measure of disability/functioning; and the London Handicap Scale (LHS)29 and the Reintegration to Normal Living Index (RNLI)30 as measures of handicap or participation. At the end of each 4 week diary period, participants completed a questionnaire asking about changes to symptoms and MS overall, medication, and relapses using questions from the cost utility study.21
Data quality
Data quality was assessed by examining response rates, missing items, and score distributions.
Reliability
Both internal and test-retest reliability were assessed. The internal structure of the MSSID was examined using principal components analysis with varimax rotation on three randomly selected days, one from each 4 week diary period. Two sets of analyses were performed, one for the symptom items (Q2) and one for the impact items (Q3–Q6). Factors were entered into the rotation if they had an eigenvalue of 1 or more, and loadings of 0.4 and above were used to determine which items belonged to which factor.31 The item content of the resulting factors were independently reviewed by a consultant neurologist (HF) to ensure they made clinical sense. Factor scores for each day were calculated by summing the items in each factor and then dividing by the total number of items completed within the factor that day. Higher scores indicated better functioning.
The internal reliability of the resulting factors was assessed using Cronbach’s alpha32 and item total correlations on three randomly selected days, one from each month. Reliability was deemed adequate if Cronbach’s alphas were >0.824 and item total correlations >0.2.23
The test-retest reliability of the resulting factors was assessed in two ways using the intraclass correlation coefficient (ICC).33 Firstly, the ICC was calculated between median week 1 and week 4 factor scores in stable respondents—that is, those whose symptoms did not change and who did not have a relapse during the first 4 weeks (n = 26). Secondly, the ICC was calculated between factor scores on the first 2 days of each 4 week diary period when the respondent indicated that their health had not changed (n = 61). ICCs between 0.70 and 0.89 were deemed adequate for group comparisons, while ICCs of 0.9 and above were deemed adequate for individual comparisons.24
Validity
Both discriminant and convergent construct validity were assessed using appropriate summary indicators of the longitudinal diary data.34 To provide a summary indicator of the impact of symptoms on daily activities (factors 1 and 2) and the impact of MS overall (factor 3), median factor scores were used. The median was chosen rather than the mean as it was expected that the median would be less influenced by missing data, and it was not expected that an individual’s scores over time would be normally distributed.
Previous research has demonstrated that people with progressive MS experience greater levels of impairment and disability than other disease course groups,35 and that people who have had a relapse experience worse health status than those who have not.21 Therefore, it was hypothesised that there would be statistically significant differences in median factor scores between those with relapsing remitting or benign MS v those with progressive MS, between those who walked unaided v those who used a walking aid and between those who experienced a relapse during the study v those who did not. In each case it was expected that for patients with progressive MS, those who used a walking aid and those who had a relapse would have worse (lower) median factor scores.
The convergent validity of the MSSID was examined by exploring correlations between the median factor scores during the last 4 week diary period and single point instruments collected at the end of the study. Correlations <0.4 were considered small, between 0.41 and 0.7 moderate, and 0.71 and above high.9
Responsiveness
Effect sizes of the change in median MSSID scores, LMSQoL, and MSQol-54 scores between week 1 and week 4 were calculated separately for those who indicated that their MS had got better (n = 14) and those who indicated that their MS had got worse (n = 20) in the first 4 weeks of the study.36 Effect sizes of 0.2 were considered to be small, 0.5 moderate, and 0.8 or more large.37 It was expected that effect sizes for the MSSID scores and MSQol-54 would be positive for those who got better and negative for those who got worse. The reverse pattern was expected for the LMSQoL because higher scores indicate worse functioning.
Statistical analyses
The differences in continuous variables (for example, MSSID scores) were assessed using Mann-Whitney U tests. Differences in categorical variables were assessed using the 2 test. Relationships between MSSID factor scores and single point instruments were assessed using Spearman’s rank correlations. All analyses were carried out in SPSS for Windows (version 9).
RESULTS
Changes made to the original diary
A number of changes were made to the original diary following the developmental stages. Seven symptom items were added to reflect issues that were raised in focus groups and interviews (for example "difficulties thinking clearly or forgetting things" and "feeling frustrated"). Some items were reworded to make them more interpretable to respondents (for example "altered sensations" was changed to "altered or loss of sensation"). The number of response options of some items was increased to better reflect the variation in symptom impact and health perceptions (for example, the response options for Q6 were reworded and increased from three to six). Finally, an open ended question (Q8) was added to allow respondents to record any additional issues or events that happened.
Sample characteristics
Sixty three people (82%) returned all three diaries; these were classed as respondents and used in subsequent analyses. Eighty one percent were women, the average age was 46 years, and they had been diagnosed with MS for an average of 11 years (table 1). Seventy three percent were currently not in paid work and 33% had experienced a relapse during the study. There were no statistically significant differences in age, gender, length of time since diagnosis, or clinical course between respondents recruited from either centre, thus the two samples were analysed together.
Table 1 Clinical and demographic characteristics of the sample
Data quality
Across the entire 12 weeks, 7% of MSSID items were missing, ranging from 5.8% to 7.6% across items. Three quarters of respondents completed at least 93% of the items and all respondents completed at least 50% of the items. Three symptom impact items (Q2) "weakness or heaviness in your legs", "problems with your bladder and bowel", and "feeling frustrated", had floor effects, with 20% or more person days rated as "a lot".
Internal reliability
The factor analysis of symptom impact items (Q2) revealed three factors that explained between 62% and 68% of the total variance. However, one factor contained only two items ("blurred, double, or shaky vision" and "coughing or choking when eating and drinking") and performed poorly in the internal reliability analyses, thus it was not retained. The remaining two symptom impact factors measured "mobility" and "fatigue". The factor analysis of the overall impact items (Q3–Q6) formed one factor that explained between 66% and 79% of the total variance and measured the overall impact of MS. For the three retained factors, Cronbach’s alpha exceeded or approached 0.8 and item total correlations exceeded 0.2 (table 2).
Table 2 Internal and test–retest reliability analysis of the MSSID
Test–retest reliability
For both weekly factor median scores and daily factor score comparisons, the ICC for fatigue and overall impact exceeded 0.7 and for mobility, exceeded 0.9 (table 2).
Discriminant validity
There were statistically significant differences in the overall impact scores between all the clinical groups in the expected direction (table 3). For mobility, there were statistically significant differences between all clinical groups in the expected direction with the exception of the relapse/no relapse comparison. For fatigue there were statistically significant differences between all clinical groups in the expected direction with the exception of the relapsing remitting/benign v progressive MS comparison. There were no statistically significant differences in the median factor scores between those recruited from either centre (data available on request).
Table 3 Discriminant validity analysis of the MSSID
Convergent validity
Table 4 shows the Spearman’s rank correlation coefficients between median factor scores during the last four weeks and the single point instruments collected at the end of the study. Mobility showed moderate correlations with the MSQol-54 physical and social functioning scales and with the RNLI. Fatigue showed moderate correlations with the MSQol-54 energy and emotional wellbeing scales and the RNLI. Overall impact of MS showed moderate correlations with the RNLI, the LMSQoL, MSQol-54 social function, and overall quality of life. The factor scores were all highly related to each other and were more highly related to each other than to the single point instruments. Fatigue and mobility were more highly related to each other than to overall impact of MS.
Table 4 Spearman’s rank correlations between median MSSID factor scores for weeks 9–12 and single point instruments collected at week 12
Responsiveness
Effect sizes for those who got better were moderate for MSSID scores and small for the MSQol-54 and LMSQol scores with one exception (emotional role limitations) (table 5). Effect sizes for those who got worse were small for the MSSID, MSQoL-54, and LMSQoL scores. The effect sizes were in the expected direction for MSSID and LMSQoL scores, but did not show a consistent pattern for MSQoL-54 scores.
Table 5 Effect sizes for median MSSID scores, MSQol-54, and LMSQoL scores for those who got better or worse in the first 4 weeks of the study
DISCUSSION
This study has provided initial support for the reliability, validity, and responsiveness of the (MSSID). Response rates were similar but the rates of missing items were higher compared with other studies using the MSQol-54, the LMSQoL, and the MSIS-29.8,9,27 However, response rates and rates of missing data were similar to other diary studies,18,22 providing initial evidence for the diary’s feasibility and acceptability. None of the items had ceiling effects and only three items had floor effects. These three items, "weakness or heaviness in your legs", "problems with your bladder and bowel", and "feeling frustrated" were retained because the developmental work indicated that they are both common and important problems for people with MS. Further work will evaluate whether excluding these items has an impact on the psychometric properties of the MSSID.
The majority of items in the MSSID focus on the physical symptoms of MS and their impact on daily activities, which is similar to the item content of the MSIS-29.9 However, like the MSIS-29, it also includes important items that address the emotional impact of the condition. The MSSID consists of three factors that measure mobility, the cognitive and emotional aspects of fatigue, and the overall impact of MS. These factors are internally reliable and their test–retest reliability was adequate for group comparisons for weekly mobility, fatigue, and overall impact scores, and individual comparisons for daily mobility and fatigue scores. The MSSID was able to discriminate between people with progressive and relapsing remitting MS, between those who used a walking aid and those who did not, and between those who had recently experienced a relapse and those who had not. It demonstrated expected moderate associations with appropriate scales on the MSQoL-54, the LMSQoL, the Barthel Index, the LHS, and the RNLI. Although there was some overlap in the constructs measured by the factor scores, they assess the impact of mobility, fatigue, and MS as a whole on daily activities.
Compared with the MSQoL-54 and the LMSQoL, the responsiveness of the median weekly MSSID scores was similar or better. The MSSID was better able to detect short term improvements in functioning than worsening, which may be attributable to the floor effects in the three items. It was also more responsive to short term improvements in functioning than the MSQoL-54 and the LMSQoL. This is in contrast to a previous study that found an asthma diary to have lower responsiveness compared with an asthma questionnaire. The better short term responsiveness of the MSSID may be because it was explicitly designed to detect short term changes in functioning, whereas the MSQoL-54 and the LMSQoL were not. The MSSID thus provides a valuable complement to currently existing instruments in its measurement of short term changes and day to day fluctuations in functioning.
The sample size used in the study was small compared with those used to develop the LMSQoL, the MSIS-29, and the MSQoL-54.8,9,26 However, it was expected to be sufficient to provide accurate estimates of the psychometric properties of the MSSID because sample sizes as low as 20 for reliability estimates and 40 for validity estimates did not alter the psychometric interpretation of the MSIS-29 compared with a sample size of 713.38 The sample was recruited from outpatient clinics, and a third of respondents experienced a relapse during the study, reflecting previous research that people with MS attending outpatient clinics are more likely to experience relapses or have greater disability for their age than community based populations.39 Therefore, further testing of the psychometric properties of the MSSID in a community based sample of people with MS would be valuable.
This study suggests that the MSSID could be used in clinical trials of both disease modifying drugs and interventions to manage MS symptoms. The MSSID may be a useful tool to monitor the occurrence of relapses and measure their impact in a psychometrically robust way. The MSSID is likely to require more resources to collect and manage the data and further research is required to assess the feasibility of its use in the context of a randomised controlled trial. However, as symptom diaries are sometimes used in current trials of MS therapies15 but are rarely psychometrically evaluated, the MSSID represents a significant advance in this respect.
The MSSID may also be useful in routine clinical practice in providing the patient with a prompt to inform clinicians about new symptoms or a worsening of existing symptoms, thus reducing the uncertainty around the occurrence of a relapse.16 It can also provide a graphical display of changes in the impact of symptoms over time and may be useful to clinicians in monitoring responses to treatment. Further work is needed to explore the most appropriate and feasible method of presenting MSSID data to clinicians and to examine clinicians’ and patients’ views of the utility of this data.
CONCLUSIONS
This study provided initial evidence that the MSSID is reliable, valid and responsive to change. It may provide a useful complement to currently available instruments to measure the outcomes of MS within clinical trials or clinical practice. Further research is needed to explore its feasibility and utility to clinicians in randomised controlled clinical trials and within routine clinical practice.
ACKNOWLEDGEMENTS
The authors would like to thank A Jacoby for permission to modify the diary from the cost utility study and her comments on the study design. We would also like to thank C Young, M Boggild, and M Johnson for their comments on the study design and for access to and recruitment of people with MS to the study. D Greenwood and S Walters provided statistical advice. Thanks also go to all the people who gave their time to participate in the research. J Greenhalgh was an MRC Training Fellow in Health Services Research 1997–2001.
REFERENCES
Rothwell PM. Quality of life in multiple sclerosis. J Neurol Neurosurg Psychiatry 1998;65:433.
Devinsky O. Outcome research in neurology: incorporating health related quality of life. Ann Neurol 1995;37:141–2.
Rothwell PM, McDowell Z, Wong CK, et al. Doctors and patients don’t agree: Cross sectional study of patients’ and doctors’ perceptions and assessments of disability in multiple sclerosis. BMJ 1997;314:1580–3.
Hoogervorst E, Eikelenboom M, Uitdehaag BMJ, et al. One year changes in disability in multiple sclerosis: neurological examination compared with patient self report. J Neurol Neurosurg Psychiatry 2003;74:439–42.
Cheng EM, Hays RD, Myers LW, et al. Factors related to agreement between self-reported and conventional Expanded Disability Status Scale (EDSS) scores. Mult Scler 2001;7:405–10.
Nortvedt MW, Riise T, Myhr KM, et al. Type I interferons and the quality of life of multiple sclerosis patients. Results from a clinical trial on interferon alfa-2a. Mult Scler 1999;5:317–22.
Greenhalgh J, Long A, Brettle AJ, et al. Reviewing and selecting outcome measures for use in routine practice. J Eval Clin Pract 1998;4:339–50.
Ford HL, Gerry E, Tennant A, et al. Developing a disease-specific quality of life measure for people with multiple sclerosis. Clinl Rehabil 2001;15:247–58.
Hobart JC, Lamping DL, Fitzpatrick R. The Multiple Sclerosis Impact Scale (MSIS-29) a new patient based outcome measure. Brain 2001;124:962–73.
Riazi A, Hobart JC, Lamping DL, et al. Multiple Sclerosis Impact Scale (MSIS-29): reliability and validity in hospital based samples. J Neurol Neurosurg Psychiatry 2002;73:701–4.
Miller C. The lived experience of multiple sclerosis. J Neurosci Nurs 1997;29:294–304.
Koopman W, Schweitzer A. The journey to multiple sclerosis: a qualitative study. J Neurosci Nurs 1999;31:17–26.
Mullins LL, Cote MP, Fuemmeler BF, et al. Illness intrusiveness, uncertainty and distress in individuals with multiple sclerosis. Rehabil Psychol 2001;46:139–53.
Kroencke D, Denny D, Lynch SG. Depression during exacerbations in multiple sclerosis: the importance of uncertainty. Mult Scler 2001;7:237–42.
Lui C, Blumhardt LD. Assessing relapses in treatment trials of relapsing and remitting multiple sclerosis: can we do better? Mult Scler 1999;5:22–8.
Baker L. Sense making in multiple sclerosis: the information needs of people with MS during an acute exacerbation. Qual Health Res 1998;8:106–20.
Sandha GS, Hunt RH, Van Zanten SJOV. A systematic overview of the use of diary cards, quality-of-life questionnaires, and psychometric tests in treatment trials of Helicobacter pylori-positive and -negative non-ulcer dyspepsia. Scand J Gastroenterol 1999;34:244–9.
Fayers P. MRC quality of life studies using a daily diary card—practical lessons learned from cancer trials. Qual Life Res 1995;4:343–52.
Juniper EF, O’Byrne PM, Ferrie PJ, et al. Measuring asthma control. Clinic questionnaire or daily diary? Am J Respir Crit Care Med 2000;162:1330–4.
Verbrugge LM, Balaban DJ. Patterns of change in disability and well being. Med Care 1989;27 (Suppl 3) :S128–47.
Parkin D, Jacoby A, McNamee P, et al. Treatment of multiple sclerosis with interferon beta: An appraisal of cost-effectiveness and quality of life. J Neurol Neurosurg Psychiatry 2000;68:144–9.
Verbrugge LM. Health diaries. Med Care 1980;18:73–95.
Streiner D, Norman G. Health measurement scales: a practical guide to their development and use, 2nd ed. Oxford: Oxford University Press, 1996.
Nunally J, Bernstein IH. Psychometric theory, 3rd ed. New York: McGraw-Hill, 1994.
Ericsson KA, Simon HA. Verbal reports as data. Psychol Rev 1980;87:215–51.
Vickrey BG, Hays RD, Harooni R, et al. A health-related quality of life measure for multiple sclerosis. Qual Life Res 1995;4:187–206.
Freeman JA, Hobart JC, Thompson AJ. Does adding MS-specific items to a generic measure (the SF-36) improve measurement? Neurology 2001;57:68–74.
Mahoney FI, Barthel DW. Functional evaluation: the Barthel Index. Md Med 1965;14:61–5.
Harwood RH, Rogers A, Dickinson E, et al. Measuring handicap: the London Handicap Scale, a new outcome measure for chronic disease. Qual Health Care 1994;3:11–16.
Wood-Dauphinee S, Opzoomer MA, Williams JI, et al. Assessment of global function: the Reintegration to Normal Living Index. Arch Phys Med Rehabil 1987;69:583–90.
Child D. Essentials of factor analysis, 2nd ed. London: Cassell, 1990.
Cronbach IJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297–334.
Armitage P, Berry G, Matthews JNS. Statistical methods in medical research, 4th ed. Oxford: Blackwell, 2002.
Matthews JNS, Altman DG, Campbell MJ, et al. Analysis of serial measurements in medical research. BMJ 1990;30:230–5.
Ford HL, Gerry E, Johnson MH, et al. Health status and quality of life of people with multiple sclerosis. Disabil Rehabil 2001;23:516–21.
Kazis L, Anderson J, Meenan R. Effect sizes for interpreting changes in health status. Med Care 1989;27 (Suppl 3):S178–89.
Cohen J. Statistical power analysis for the behavioural sciences. New York: Academic Press, 1977.
Hobart JC. What sample size for reliability and validity? Qual Life Res 2002;11:636 (Abstract).
Nelson LM, Franklin GM, Hanman RF, et al. Referral bias in multiple sclerosis research. J Clin Epidemiol 1988;41:187–92.(J Greenhalgh1, H Ford2, A)