Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators – McGill et al, 2020 – and another take on ‘systems’ and/vs ‘complexity’

A good approach to looking at how ‘complex’ is actually conceptualise and used in practice (TL:DR – it’s rather confused and more talked about than practised, and practice isn’t clear)

source:

Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators

Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators

PLOS

Abstract

Background

Public health evaluation methods have been criticized for being overly reductionist and failing to generate suitable evidence for public health decision-making. A “complex systems approach” has been advocated to account for real world complexity. Qualitative methods may be well suited to understanding change in complex social environments, but guidance on applying a complex systems approach to inform qualitative research remains limited and underdeveloped. This systematic review aims to analyze published examples of process evaluations that utilize qualitative methods that involve a complex systems perspective and proposes a framework for qualitative complex system process evaluations.

Methods and findings

We conducted a systematic search to identify complex system process evaluations that involve qualitative methods by searching electronic databases from January 1, 2014–September 30, 2019 (Scopus, MEDLINE, Web of Science), citation searching, and expert consultations. Process evaluations were included if they self-identified as taking a systems- or complexity-oriented approach, integrated qualitative methods, reported empirical findings, and evaluated public health interventions. Two reviewers independently assessed each study to identify concepts associated with the systems thinking and complexity science traditions. Twenty-one unique studies were identified evaluating a wide range of public health interventions in, for example, urban planning, sexual health, violence prevention, substance use, and community transformation. Evaluations were conducted in settings such as schools, workplaces, and neighborhoods in 13 different countries (9 high-income and 4 middle-income). All reported some utilization of complex systems concepts in the analysis of qualitative data. In 14 evaluations, the consideration of complex systems influenced intervention design, evaluation planning, or fieldwork. The identified studies used systems concepts to depict and describe a system at one point in time. Only 4 evaluations explicitly utilized a range of complexity concepts to assess changes within the system resulting from, or co-occurring with, intervention implementation over time. Limitations to our approach are including only English-language papers, reliance on study authors reporting their utilization of complex systems concepts, and subjective judgment from the reviewers relating to which concepts featured in each study.

Conclusion

This study found no consensus on what bringing a complex systems perspective to public health process evaluations with qualitative methods looks like in practice and that many studies of this nature describe static systems at a single time point. We suggest future studies use a 2-phase framework for qualitative process evaluations that seek to assess changes over time from a complex systems perspective. The first phase involves producing a description of the system and identifying hypotheses about how the system may change in response to the intervention. The second phase involves following the pathway of emergent findings in an adaptive evaluation approach.

Author summary

Why was this study done?

  • Process evaluations are used in public health to understand how and why an intervention works (or does not work), for which population groups, and in which settings.
  • Process evaluations often use qualitative methods—such as interviewing people and observing people in their daily and work routines—in order to draw their conclusions.
  • Researchers in public health have contended that we need to do research in a manner that considers the broader system in which policies and interventions take place—something we call a “complex systems perspective.”
  • To date and to our knowledge, there is no specific framework that describes how researchers can use a complex systems perspective when they conduct a process evaluation with qualitative methods.

What did the researchers do and find?

  • We conducted a systematic literature review that looked for examples of qualitative process evaluations that self-identify as using a complex systems perspective to evaluate public health interventions.
  • We found 21 different evaluations of many different types of public health interventions, including interventions to address student and employee health, sexual health, child development and safety, community empowerment, violence prevention, and substance use.
  • We found that these evaluations describe the systems in which public health efforts take place but are less effective at analyzing how changes affecting health occur within these systems.

What do these findings mean?

  • There is little evidence of a commonly shared understanding of how best to bring a complex systems perspective to process evaluations using qualitative methods, particularly, how to assess how interventions interact with a changing system.
  • We developed a 2-phase framework to guide researchers who want to apply a complex systems perspective to qualitative process evaluations.
  • This review excluded studies that do not self-identify as using a complex systems perspective so we may have missed literature that uses this perspective but not the associated terminology.

full paper in source:

Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators

The ‘complexity’ bit is a cogent attempt at a hard distinction between ‘systems thinking’ and ‘complexity’, but becomes a very soft distinction (I would say) – particularly because they are looking at that practical application:

“Research into complex systems takes place across academic disciplines and has roots in both systems thinking and complexity science. Although often grouped together because of some conceptual similarities, systems thinking and complexity science can be considered as distinct yet overlapping traditions [16,17]. Systems thinking may be best described as an orientation that prompts researchers to take a holistic, rather than reductionist view, of phenomena and study them in the context of their real-world systems that are open to and interact with surrounding systems. Systems thinking draws on theories, concepts, and methods from a range of disciplinary fields [18]. Complexity science, on the other hand, is more strongly rooted in the mathematical sciences and has drawn on complexity theory, which emphasizes uncertainty and nonlinearity, to create and refine specific methodological approaches to modeling complex systems in order to estimate and predict their emergent behavior over time. Systems thinking prompts researchers and practitioners to consider the boundaries of the system they are studying or in which they are working [19] and places an emphasis on the interactions and relationships between system elements and the system with its broader environment [1,6]. Further applying concepts from complexity science prompts a consideration of how those interactions create nonlinear chains of cause and effect, are unpredictable, unfold overtime, and give rise to system-level emergent outcomes [20].”