How do we evaluate systems change?
How do we evaluate systems change?
When I think about systems change, let alone evaluating it, I sometimes don’t know where to start.
Let’s consider the system of racial inequity. As Race Forward describes, racism is upheld in institutional and structural racism that consists of policies, practices, and resource flows that unjustly benefit white people. For instance, the typical white family has eight times the wealth of a typical Black family and more than families of other racial/ethnic groups, driven in a large part by housing segregation, differences in intergenerational wealth, and other economic policies.
But even when policies change, attitudes and mental models still affect systems. Racist beliefs, or even implicit bias or unrealized intergenerational trauma, can lead to discriminatory actions in housing, education, policing, and beyond.
And yet, changing mental models and attitudes alone is not enough. For example, an implicit bias training for NYPD failed to reduce discriminatory practices. Further shifts in power dynamics, policing culture, and policies continue to be necessary.
Systems change requires change in all of these areas: what is within and between people, and within and between institutions.
The Water of Systems Change series by FSG connects attitude shifts, power dynamics, and policies/infrastructure all together in their Six Conditions of Systems Change model:
From FSG (John Kania, Mark Kramer, Peter Senge), 2018, Water of Systems Change
This model resonates a lot with me, especially how all of these pieces need to be in place for a system to really change, and that a change in one aspect can accelerate or hinder changes in others. There is no hierarchy to which is more important, just to which pieces are more explicit or implicit.
What does this mean for evaluation?
At MXM we've been thinking a lot about the implications for evaluation. We partner with one organization, Coalition of Racial Equity in Mental Health (CORE Mental Health), that works to create systems change around racial inequities in mental health. Their work addresses knowledge and self-efficacy (which contribute to mental models) through training, relationships through learning communities, and practices through support and funding - among many other activities. Other programs may try to address specific policies or practices, but not have an explicit goal for mental models - or vice versa.
Systems change efforts are often messier than intended, with outcomes emerging years after a project has ended. One approach we use for partners like CORE Mental Health is to use a logic model and measurement framework for the systems change component that are more linear and shorter term. We complement these with complexity-focused methods like outcome harvesting for longer-term, emergent outcomes where the pathway of change may be uncertain. It takes time to fine tune an evaluation plan that leaves space for complexity and uncertainty, while also ensuring that programs and coalitions have concrete accountability for their constituents and their learning needs met.
What are you learning?
Have you worked on evaluating systems change efforts? What's something you've learned or are chewing on in your work right now?