InSupplyHealth

When to use an Outcome Harvesting evaluation approach: Lessons from measurement of community health supply chain models in four Kenya counties

Counties in the arid/semi-arid lands (ASAL) of Kenya host remote, hard-to-reach, and migratory populations and continue to have low modern contraceptive use prevalence rates despite improvements nationally. The Supply Chain Alternatives for the Last Mile Equity (SCALE) project, implemented by inSupply Health in partnership with Kenya’s Ministry of Health and with funding from the Bill & Melinda Gates Foundation (BMGF), aims to address inequities in access to health commodities in four of these underserved, remote communities of Kenya (Samburu, Turkana, Wajir, and Mandera). Our objective was to develop sustainable, scalable, and community-based distribution models to improve supply chains.

The SCALE project’s interventions were closely linked to those of two other projects: Data Use Project in Kenya and Tanzania also implemented by inSupply Health and the Nomadic Health Program (NHP) implemented by Save the Children.

Outcome Harvesting Evaluation Approach

The SCALE project used an Outcome Harvesting (OH) evaluation approach to generate evidence around intervention components that lead to sustainable community health models. OH is a complexity-awareness-monitoring approach that uses participatory methods to identify intended and unintended outcomes of a program or intervention. OH works best in complex environments and for programs where the paths of change are dynamic and there is no clear relationship between interventions and outcomes.

OH collects evidence on the outcomes and then works backwards and determines whether and how an intervention has contributed to the identified outcomes. There are four key players in an outcome harvesting:
Change agents who influence an outcome; 

Social actors who are individuals that change as a result of an intervention;

Harvesters are the people responsible for managing the OH activity (evaluators);

Harvest users or stakeholders that need the findings to make decisions.

Another group necessary for an OH evaluation are substantiators who validate the outcomes. Substantiators can be both change agents or social actors.

Reasons for choosing OH for evaluating SCALE interventions

  • While the project undertook an impressively long list of activities, we wanted to identify and describe the effects of our interventions. In other words, the intended and unintended outcomes of cStock and IMPACT Teams on community health programs. 
  • Both interventions (cStock and IMPACT Teams) were multifaceted with no clear way to differentiate which components of the interventions led to outcomes.
  • SCALE was being implemented in close collaboration with the NHP project. Both projects were funded by BMGF, were working in similar geographies, and leveraged on each other’s support. The close collaboration made it complex and difficult to parse out individual impacts; therefore, the projects were perfect candidates for OH.

Process

We started by developing our research questions. The small team of harvesters began by collating all existing evidence including reports, meeting notes, publications, reports from the developmental evaluation (DE) activity, and other resources related to the project interventions. We engaged with change agents (our implementing team) to get more information on what was found in the document review and glean additional context. The harvesters drafted outcome descriptions, which were then substantiated by county stakeholders. Substantiators agreed with most of the outcomes. Lastly, the harvesters analyzed and grouped the substantiated outcomes by research question. Our report provided recommendations to donors, county health officials, and other implementers.

Source: https://www.betterevaluation.org/en/plan/approach/outcome_harvesting

Timeline

The OH process lasted 4 months: In June 2021 we began the harvest design, in August 2021  we had the substantiation meeting, and in September 2021 we started the dissemination of final results.

Findings

The OH determined that cStock and IMPACT teams were effective in increasing access to community health services. Based on these OH findings, we recommended that cStock be scaled up to additional geographical areas and levels of the health system. We also found that when implemented with cStock, IMPACT Teams provide a sustainable way to utilize the data produced by the commodity tracking tool. 

Lessons Learned

During our implementation of OH, we learned the following: 

  • Proper documentation of the outcomes by the change agents is critical for OH as it minimizes the back and forths by the harvesters seeking clarifications while consolidating the outcome descriptions.  
  • We found the validation of results by substantiators to be one of the most beneficial components of OH because it provided the implementing team and harvesters with the opportunity to hear directly from social actors whether they agreed or disagreed with the documented outcomes. We heard directly from those who we hoped to have an influence on and were able to back up their interpretations with data.
  • Having the right combination of substantiators is valuable to the process. Substantiators must be able to speak to in-depth details about the project implementation and have insights into execution and success or failure of project activities. 
  • Sharing outcome drafts with participants before the substantiation meeting is beneficial. It gives participants time to consult their additional data sources, if needed, to substantiate the outcomes. 
  • During OH, the number of total outcomes can quickly become unwieldy. To ensure in-depth review of each outcome and contribution by all teams, it is important to balance the number of outcomes to review among the substantiators. 
  • One key feature of the substantiation meeting was minimal presence of project staff during the meeting. This creates a favorable and comfortable environment for substantiators to speak freely without fear of recourse from project teams.
  • The process of OH sheds light on the processes of change and how a project or intervention contributes to those processes. Outcome descriptions can be difficult to write, take time, and require a lot of follow up with actors to fill in gaps from the document review.

Conclusion

The use of OH as an evaluation method can be valuable for demonstrating the effects of complex and multifaceted interventions. The participatory nature of the OH method allows input and validation from stakeholders most affected by the interventions. Furthermore, OH’s ability to measure outcomes in complex environments and with multi-faceted interventions are especially important in global health settings where projects are often fragmented or where there are multiple factors at play with one group of end users. Traditional monitoring and evaluation methods make it more difficult to tease out intervention effects. The SCALE project met these requirements and thus was a great way to implement OH and test its success.

While OH might not be considered a mixed methods approach, it provides similar benefits because some of the outcomes are backed by quantitative evidence and have qualitative context to support the findings.

inSupply Health is committed to identifying and using new and innovative methodologies to demonstrate the effects of complex, dynamic interventions. Participatory methods, such as OH and DE, provide a new opportunity for end users of global health interventions to have a say in that intervention’s effectiveness.

Read the full report here.

Watch the webinar on Outcome Harvesting and Developmental Evaluation methods here