Integrating qualitative research with trials in systematic reviews
http://www.100md.com
《英国医生杂志》
1 Evidence for Policy and Practice Information and Coordinating (EPPI) Centre, Social Science Research Unit, Institute of Education, London WC1H 0NR
Correspondence to: J Thomas j.thomas@ioe.ac.uk
An example review from public health shows how integration is possible and some potential benefits
The value of including data from different types of studies in systematic reviews of health interventions is increasingly recognised. A recent editorial accepted that qualitative research should be included in systematic reviews, but pointed to a "daunting array of theoretical and practical problems."1 This article presents an approach to combining qualitative and quantitative research in a systematic review. We describe how we used this approach in a systematic review of interventions to promote healthy eating among children, full details of which are available.2
The review framework
The review question was: "What is known about the barriers to, and facilitators of, healthy eating among children aged 4-10 years?" The specific focus of the review was fruit and vegetable intake. We searched for two types of research: controlled trials (randomised or non-randomised) that examined interventions to promote healthy eating and studies that examined children's perspectives and understandings (views studies), often by using qualitative research methods—for example, in-depth interviews and focus groups.
But will she eat her greens?
Credit: PAUL DARRAH/REX
We used conventional systematic review methods: sensitive searching, systematic screening, and independent quality assessment. These methods found 33 trials and eight qualitative studies that met our prespecified inclusion criteria.
We assessed studies for quality and reliability according to standards for their specific study types; they were then synthesised individually by using methods appropriate to the study. We conducted a meta-analysis with the data extracted from trials, used qualitative methods to synthesise the textual data extracted from the qualitative studies, and then integrated the findings from the qualitative synthesis with those from the meta-analysis. This gave us one review with three syntheses (fig 1).
Fig 1 Stages of the review
Quality assessment
We maintained the key principles of avoiding bias and maximising transparency and accountability when conducting a systematic review. Both types of study went through a stage of quality assessment with two reviewers working independently and then meeting to discuss their findings. We used different tools for the different types of studies, building on recent developmental work and established consensus on quality assessment for both experimental studies3-6 and qualitative research.7-11 The studies were assessed in terms of reporting quality, internal validity or reliability, and, for qualitative studies, the extent to which the findings were rooted in children's perspectives (box).
We judged 21 of the 33 trials to be sufficiently reliable to enter the meta-analysis. Five of the eight qualitative studies met nine or more of the 12 quality criteria. The remaining three met six or fewer criteria. We conducted a sensitivity analysis and found that the results of these three studies did not contradict those from studies of a higher quality. The synthesis would have come to the same conclusions with or without their inclusion. In future, we have decided to exclude poorer quality studies from the synthesis and are conducting methodological work to assess the impact this has on the findings of the review.
Synthesis 1: meta-analysis of data from trials
In the first synthesis we carried out a traditional meta-analysis and pooled the effect sizes on six outcomes. We explored heterogeneity by carrying out subgroup analyses on a limited range of categories that we had specified in advance. Combining the results of the trials using a random effects model we found that, on average, the interventions described in the trials were able to increase children's fruit and vegetable consumption by about half a portion a day.
There was great variability between the studies. For example, one intervention was able to increase consumption by nearly two portions a day, while most of the others did not achieve one portion. Since all but two of the studies were evaluating different interventions, the summary statistic seems to conceal more than it reveals. We were unable to explain the statistical heterogeneity using prespecified categories covering study quality, study design, setting, and type of intervention.
Synthesis 2: synthesis of qualitative studies
The data for the second synthesis were in text form. In order to synthesise these, we copied the authors' findings verbatim into NVivo software12 and then followed guidelines for the thematic analysis of textual data collected in primary research. The aim of the analysis was to infer barriers to, and facilitators of, healthy eating and ideas for effective interventions from children's views. We examined the findings of each study in turn and assigned codes to describe relevant sentences or paragraphs—for example, one code was "children prefer fruit to vegetables." We then looked for similarities and differences between the codes to organise these into a hierarchical tree structure centred on children's understandings of healthy eating and the factors, in their views, that influence the food they eat.
In the next stage of the synthesis, three reviewers independently examined the descriptive themes and their associated data in the light of the review question to infer barriers, facilitators, and implied recommendations for developing interventions. The reviewers then met to discuss their findings and to develop a set of more abstract analytical themes.
As an example, one of the themes was that children do not see their personal health as their responsibility but that of their parents. Children do not regard purchasing fruit for health reasons as a legitimate use of their pocket money. Again, that is the job of parents. Children prioritise taste over health; as one child said, "Everything that is healthy tastes awful." This theme suggested that future health promotion interventions should promote fruit and vegetables as tasty rather than healthy and any emphasis on health messages should be minimised. A second theme was that children distinguish between fruit and vegetables—they do not regard them as the same kind of food. The implication here is that fruit and vegetables should not be promoted in the same way within the same intervention. Although the five a day catchphrase might be appropriate for adults, it could be a disincentive for children.
Criteria for assessing quality
Quantitative studies (controlled trials)
Provision of data on outcomes before and after the intervention
Provision of data on all outcomes measured
Use of an equivalent control or comparison group
Qualitative studies
Quality of reporting (5 items)
Sufficiency of strategies for establishing reliability or validity (4 items)
Extent to which study findings were rooted in children's perspectives (3 items)
Synthesis 3: bringing the two sets of studies together
In the third synthesis we used the results of our qualitative synthesis to combine the findings from the controlled trials and qualitative studies. We then devised a matrix that juxtaposed the barriers, facilitators, and implied recommendations against the actual interventions that had been implemented and evaluated. Since we could not know the outcome of the qualitative synthesis in advance, we had to go back to the original interventions evaluated in the trials to identify those that built on the barriers and facilitators suggested by the children. This comparative analysis was guided by three questions:
Which interventions match recommendations derived from children's views and experiences?
Which recommendations have yet to be tried in soundly evaluated interventions?
Do those interventions that match recommendations show bigger effect sizes or explain heterogeneity?
The table shows an example from the matrix. We used the good quality trials to assess whether the evidence of effectiveness supported or contradicted the children's views and to identify gaps in the evidence. The interventions that had not been evaluated well but were identified as building on a potential facilitator were recommended for more rigorous evaluation.
Example of the synthesis matrix
For the recommendation that fruit and vegetables should not be promoted in the same way, we found no sound trials, so we identified a research gap. We found five good quality trials relevant to the second recommendation—reducing the emphasis on health messages. Two of these provided results on the same outcome, so we were able to conduct a statistical subgroup analysis dividing the studies into those that emphasised health messages and those that did not. (Since the original meta-analysis in synthesis 1 suggested that interventions targeting physical activity as well as healthy eating were qualitatively different from those that did not, the subgroup analysis excluded the interventions with a physical activity component.)
Summary points
Qualitative studies improve understanding of the views of the target group of an intervention
Uncertainty exists about how to include qualitative research within systematic reviews
A three stage method is described to integrate qualitative studies with controlled trials in one systematic review
Integration of the two types of studies can identify ways to improve interventions and their implementation
Figure 2 shows data from the trials in this subgroup analysis. The only two studies to increase vegetable consumption by more than 0.4 portion a day were the two that had little or no emphasis on health messages. We found that highly significant heterogeneity was explained by this subdivision.13 As with any exploration of heterogeneity, this was an exercise in hypothesis generation. Since we were dealing with small numbers of studies, our conclusions had to be cautious.
Fig 2 Increase in consumption of fruit and vegetables in trials with data on health emphasis
Discussion
Integrating different types of study and data within the same review is one of the key challenges facing systematic reviewers today. We have discussed the issues raised by this at greater length elsewhere.14-15 The main importance of this challenge is that the conclusions of reviews may be substantially altered by the inclusion of qualitative data, which are more likely to reflect the experiences of the target groups for intervention. This in turn could lead to the development of more appropriate and effective interventions.
However, the approach outlined here raises some methodological and conceptual challenges. Firstly, it challenges the notion that subgroup analyses should always be specified before the review. The inductive approach used in the thematic analysis of data from the qualitative studies meant that our categories for subgroup analysis could not be defined in advance. Secondly, the use of children's views to structure the final synthesis challenges traditional notions of who experts are and what constitutes expert opinion. Thirdly, the method is dependent on the judgment of reviewers when evaluating the extent to which an intervention meets a recommendation from the qualitative synthesis. Decisions also have to be made when the findings of the two syntheses conflict or when different parts of the matrix suggest contrary approaches.
The technique presented here breaks new ground in review methodology, offering an alternative to Bayesian methods for combining different types of studies in systematic reviews.16 Conceptually, the method allows the integration of quantitative estimates of benefit and harm with qualitative understanding from people's lives. The insights gained from the synthesis of qualitative studies allows exploration of statistical heterogeneity in ways that it would be difficult to imagine in advance. More work is needed to develop the method and test its relevance to different areas of health care and health promotion research.
An earlier draft of this paper was presented at the 11th Cochrane Colloquium in Barcelona, 27 October 2003.
Contributors and sources: Through a series of reviews funded by the Department of Health (England), the authors have been developing methods to synthesise different types of studies in a systematic and rigorous way. They are also involved in training and supporting other researchers who are undertaking systematic reviews in education and other areas of public policy.
Funding: The systematic review was funded by the Department of Health (England).
Competing interests: None declared.
References
Dixon-Woods M, Fitzpatrick R. Qualitative research in systematic reviews. BMJ 2001;323: 765-6.
Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, et al. Children and healthy eating: a systematic review of barriers and facilitators. London: Evidence for Policy and Practice Information and Coordinating Centre, 2003. http://eppi.ioe.ac.uk/EPPIWeb/home.aspx?page=/hp/reports/healthy_eating02/healthy_eating02.htm (accessed 25 Feb 2004).
Oakley A, Fullerton D, Holland J, Arnold S, France-Dawson M, Kelley P, McGrellis S. Sexual health education interventions for young people: A methodological review. BMJ 1995;310: 158-62.
Peersman G, Oliver S, Oakley A. EPPI-Centre review guidelines: data collections for the EPIC database. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, 1997.
Loevinsohn BP. Health education interventions in developing countries: a methodological review of published articles. Int J Epidemiol 1990;4: 788-94.
McDonald G, Sheldon B, Gillespie J. Contemporary studies of the effectiveness of social work. Br J Soc Work 1992;22: 615-43.
Boulton M, Fitzpatrick R, Swinburn C. Qualitative research in health care II: a structured review and evaluation of studies. J Eval Clin Pract 1996;2: 171-9.
Cobb AK, Hagemaster JN. Ten criteria for evaluation qualitative research proposals. J Nurs Educ 1987;26: 138-43.
Medical Sociology Group. Criteria for the evaluation of qualitative research papers. Med Sociol News 1996;22: 69-71.
Mays N, Pope C. Rigour and qualitative research. BMJ 1995;311: 109-12.
Harden A, Oakley A, Oliver S. Peer-delivered health promotion for young people: a systematic review of different study designs. Health Educ J 2001;60: 339-53.
QSR International. NVivo version 2.0. www.qsrinternational.com (accessed 25 Feb 2004).
Lipsey MW, Wilson DB. Practical meta-analysis. London: Sage Publications, 2001.
Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A. Applying systematic review methods to studies of people's views: an example from public health. J Epidemiol Community Health (in press).
Harden A, Thomas J. Methodological issues in combining diverse study types in systematic reviews. Int J Soc Res Methodol (in press).
Roberts KA, Dixon-Woods M, Fitzpatrick R, Abrams KR, Jones DR. Factors affecting uptake of childhood immunisation: a Bayesian synthesis of qualitative and quantitative evidence. Lancet 2002;360: 1596-9.(James Thomas, research of)
Correspondence to: J Thomas j.thomas@ioe.ac.uk
An example review from public health shows how integration is possible and some potential benefits
The value of including data from different types of studies in systematic reviews of health interventions is increasingly recognised. A recent editorial accepted that qualitative research should be included in systematic reviews, but pointed to a "daunting array of theoretical and practical problems."1 This article presents an approach to combining qualitative and quantitative research in a systematic review. We describe how we used this approach in a systematic review of interventions to promote healthy eating among children, full details of which are available.2
The review framework
The review question was: "What is known about the barriers to, and facilitators of, healthy eating among children aged 4-10 years?" The specific focus of the review was fruit and vegetable intake. We searched for two types of research: controlled trials (randomised or non-randomised) that examined interventions to promote healthy eating and studies that examined children's perspectives and understandings (views studies), often by using qualitative research methods—for example, in-depth interviews and focus groups.
But will she eat her greens?
Credit: PAUL DARRAH/REX
We used conventional systematic review methods: sensitive searching, systematic screening, and independent quality assessment. These methods found 33 trials and eight qualitative studies that met our prespecified inclusion criteria.
We assessed studies for quality and reliability according to standards for their specific study types; they were then synthesised individually by using methods appropriate to the study. We conducted a meta-analysis with the data extracted from trials, used qualitative methods to synthesise the textual data extracted from the qualitative studies, and then integrated the findings from the qualitative synthesis with those from the meta-analysis. This gave us one review with three syntheses (fig 1).
Fig 1 Stages of the review
Quality assessment
We maintained the key principles of avoiding bias and maximising transparency and accountability when conducting a systematic review. Both types of study went through a stage of quality assessment with two reviewers working independently and then meeting to discuss their findings. We used different tools for the different types of studies, building on recent developmental work and established consensus on quality assessment for both experimental studies3-6 and qualitative research.7-11 The studies were assessed in terms of reporting quality, internal validity or reliability, and, for qualitative studies, the extent to which the findings were rooted in children's perspectives (box).
We judged 21 of the 33 trials to be sufficiently reliable to enter the meta-analysis. Five of the eight qualitative studies met nine or more of the 12 quality criteria. The remaining three met six or fewer criteria. We conducted a sensitivity analysis and found that the results of these three studies did not contradict those from studies of a higher quality. The synthesis would have come to the same conclusions with or without their inclusion. In future, we have decided to exclude poorer quality studies from the synthesis and are conducting methodological work to assess the impact this has on the findings of the review.
Synthesis 1: meta-analysis of data from trials
In the first synthesis we carried out a traditional meta-analysis and pooled the effect sizes on six outcomes. We explored heterogeneity by carrying out subgroup analyses on a limited range of categories that we had specified in advance. Combining the results of the trials using a random effects model we found that, on average, the interventions described in the trials were able to increase children's fruit and vegetable consumption by about half a portion a day.
There was great variability between the studies. For example, one intervention was able to increase consumption by nearly two portions a day, while most of the others did not achieve one portion. Since all but two of the studies were evaluating different interventions, the summary statistic seems to conceal more than it reveals. We were unable to explain the statistical heterogeneity using prespecified categories covering study quality, study design, setting, and type of intervention.
Synthesis 2: synthesis of qualitative studies
The data for the second synthesis were in text form. In order to synthesise these, we copied the authors' findings verbatim into NVivo software12 and then followed guidelines for the thematic analysis of textual data collected in primary research. The aim of the analysis was to infer barriers to, and facilitators of, healthy eating and ideas for effective interventions from children's views. We examined the findings of each study in turn and assigned codes to describe relevant sentences or paragraphs—for example, one code was "children prefer fruit to vegetables." We then looked for similarities and differences between the codes to organise these into a hierarchical tree structure centred on children's understandings of healthy eating and the factors, in their views, that influence the food they eat.
In the next stage of the synthesis, three reviewers independently examined the descriptive themes and their associated data in the light of the review question to infer barriers, facilitators, and implied recommendations for developing interventions. The reviewers then met to discuss their findings and to develop a set of more abstract analytical themes.
As an example, one of the themes was that children do not see their personal health as their responsibility but that of their parents. Children do not regard purchasing fruit for health reasons as a legitimate use of their pocket money. Again, that is the job of parents. Children prioritise taste over health; as one child said, "Everything that is healthy tastes awful." This theme suggested that future health promotion interventions should promote fruit and vegetables as tasty rather than healthy and any emphasis on health messages should be minimised. A second theme was that children distinguish between fruit and vegetables—they do not regard them as the same kind of food. The implication here is that fruit and vegetables should not be promoted in the same way within the same intervention. Although the five a day catchphrase might be appropriate for adults, it could be a disincentive for children.
Criteria for assessing quality
Quantitative studies (controlled trials)
Provision of data on outcomes before and after the intervention
Provision of data on all outcomes measured
Use of an equivalent control or comparison group
Qualitative studies
Quality of reporting (5 items)
Sufficiency of strategies for establishing reliability or validity (4 items)
Extent to which study findings were rooted in children's perspectives (3 items)
Synthesis 3: bringing the two sets of studies together
In the third synthesis we used the results of our qualitative synthesis to combine the findings from the controlled trials and qualitative studies. We then devised a matrix that juxtaposed the barriers, facilitators, and implied recommendations against the actual interventions that had been implemented and evaluated. Since we could not know the outcome of the qualitative synthesis in advance, we had to go back to the original interventions evaluated in the trials to identify those that built on the barriers and facilitators suggested by the children. This comparative analysis was guided by three questions:
Which interventions match recommendations derived from children's views and experiences?
Which recommendations have yet to be tried in soundly evaluated interventions?
Do those interventions that match recommendations show bigger effect sizes or explain heterogeneity?
The table shows an example from the matrix. We used the good quality trials to assess whether the evidence of effectiveness supported or contradicted the children's views and to identify gaps in the evidence. The interventions that had not been evaluated well but were identified as building on a potential facilitator were recommended for more rigorous evaluation.
Example of the synthesis matrix
For the recommendation that fruit and vegetables should not be promoted in the same way, we found no sound trials, so we identified a research gap. We found five good quality trials relevant to the second recommendation—reducing the emphasis on health messages. Two of these provided results on the same outcome, so we were able to conduct a statistical subgroup analysis dividing the studies into those that emphasised health messages and those that did not. (Since the original meta-analysis in synthesis 1 suggested that interventions targeting physical activity as well as healthy eating were qualitatively different from those that did not, the subgroup analysis excluded the interventions with a physical activity component.)
Summary points
Qualitative studies improve understanding of the views of the target group of an intervention
Uncertainty exists about how to include qualitative research within systematic reviews
A three stage method is described to integrate qualitative studies with controlled trials in one systematic review
Integration of the two types of studies can identify ways to improve interventions and their implementation
Figure 2 shows data from the trials in this subgroup analysis. The only two studies to increase vegetable consumption by more than 0.4 portion a day were the two that had little or no emphasis on health messages. We found that highly significant heterogeneity was explained by this subdivision.13 As with any exploration of heterogeneity, this was an exercise in hypothesis generation. Since we were dealing with small numbers of studies, our conclusions had to be cautious.
Fig 2 Increase in consumption of fruit and vegetables in trials with data on health emphasis
Discussion
Integrating different types of study and data within the same review is one of the key challenges facing systematic reviewers today. We have discussed the issues raised by this at greater length elsewhere.14-15 The main importance of this challenge is that the conclusions of reviews may be substantially altered by the inclusion of qualitative data, which are more likely to reflect the experiences of the target groups for intervention. This in turn could lead to the development of more appropriate and effective interventions.
However, the approach outlined here raises some methodological and conceptual challenges. Firstly, it challenges the notion that subgroup analyses should always be specified before the review. The inductive approach used in the thematic analysis of data from the qualitative studies meant that our categories for subgroup analysis could not be defined in advance. Secondly, the use of children's views to structure the final synthesis challenges traditional notions of who experts are and what constitutes expert opinion. Thirdly, the method is dependent on the judgment of reviewers when evaluating the extent to which an intervention meets a recommendation from the qualitative synthesis. Decisions also have to be made when the findings of the two syntheses conflict or when different parts of the matrix suggest contrary approaches.
The technique presented here breaks new ground in review methodology, offering an alternative to Bayesian methods for combining different types of studies in systematic reviews.16 Conceptually, the method allows the integration of quantitative estimates of benefit and harm with qualitative understanding from people's lives. The insights gained from the synthesis of qualitative studies allows exploration of statistical heterogeneity in ways that it would be difficult to imagine in advance. More work is needed to develop the method and test its relevance to different areas of health care and health promotion research.
An earlier draft of this paper was presented at the 11th Cochrane Colloquium in Barcelona, 27 October 2003.
Contributors and sources: Through a series of reviews funded by the Department of Health (England), the authors have been developing methods to synthesise different types of studies in a systematic and rigorous way. They are also involved in training and supporting other researchers who are undertaking systematic reviews in education and other areas of public policy.
Funding: The systematic review was funded by the Department of Health (England).
Competing interests: None declared.
References
Dixon-Woods M, Fitzpatrick R. Qualitative research in systematic reviews. BMJ 2001;323: 765-6.
Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, et al. Children and healthy eating: a systematic review of barriers and facilitators. London: Evidence for Policy and Practice Information and Coordinating Centre, 2003. http://eppi.ioe.ac.uk/EPPIWeb/home.aspx?page=/hp/reports/healthy_eating02/healthy_eating02.htm (accessed 25 Feb 2004).
Oakley A, Fullerton D, Holland J, Arnold S, France-Dawson M, Kelley P, McGrellis S. Sexual health education interventions for young people: A methodological review. BMJ 1995;310: 158-62.
Peersman G, Oliver S, Oakley A. EPPI-Centre review guidelines: data collections for the EPIC database. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, 1997.
Loevinsohn BP. Health education interventions in developing countries: a methodological review of published articles. Int J Epidemiol 1990;4: 788-94.
McDonald G, Sheldon B, Gillespie J. Contemporary studies of the effectiveness of social work. Br J Soc Work 1992;22: 615-43.
Boulton M, Fitzpatrick R, Swinburn C. Qualitative research in health care II: a structured review and evaluation of studies. J Eval Clin Pract 1996;2: 171-9.
Cobb AK, Hagemaster JN. Ten criteria for evaluation qualitative research proposals. J Nurs Educ 1987;26: 138-43.
Medical Sociology Group. Criteria for the evaluation of qualitative research papers. Med Sociol News 1996;22: 69-71.
Mays N, Pope C. Rigour and qualitative research. BMJ 1995;311: 109-12.
Harden A, Oakley A, Oliver S. Peer-delivered health promotion for young people: a systematic review of different study designs. Health Educ J 2001;60: 339-53.
QSR International. NVivo version 2.0. www.qsrinternational.com (accessed 25 Feb 2004).
Lipsey MW, Wilson DB. Practical meta-analysis. London: Sage Publications, 2001.
Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A. Applying systematic review methods to studies of people's views: an example from public health. J Epidemiol Community Health (in press).
Harden A, Thomas J. Methodological issues in combining diverse study types in systematic reviews. Int J Soc Res Methodol (in press).
Roberts KA, Dixon-Woods M, Fitzpatrick R, Abrams KR, Jones DR. Factors affecting uptake of childhood immunisation: a Bayesian synthesis of qualitative and quantitative evidence. Lancet 2002;360: 1596-9.(James Thomas, research of)