Menu Close

Project Evaluation and Outcome Harvesting

Overview

Evaluation is an assessment of a project’s achievements against a set or expected targets. The standard evaluation processes follow the OECD criteria of Relevance, Coherence, Effectiveness, Efficiency, Impact, and Sustainability. There are other several approaches to project evaluation where objectives and target results dictate the method. This is the principle behind Utilization-Focused Evaluation (UFE) and Adaptive Management. An innovative entry into project evaluation methods is outcome harvesting where projects identify and analyse a project’s effects, identifying outcomes that are related to its interventions.

Right Track Africa (RTA) has provided evaluation services to numerous organizations and projects, using a combination of means guided by the assignment is Terms of References, as well as consultatively agreed analytical requirements for effective lesson learning reports. By using those that are intended or unintended, positive or negative, RTA has been able to support projects in developing lessons for the project teams and resource-providers (funders), and inform or advise related or subsequent initiatives.

RTA’s Evaluation and Outcome Harvesting Projects

End-line evaluation of Action Aid’s TBGEK’s project (Nov 2019 to Mar 2020)

Through the support of the Peoples’ Postcode Lottery (UK), Action Aid International Kenya (AAIK), in partnership with its local partners, has been implementing a project (“Tackling Barriers to Girls’ Education (TBGEK)”) – over a period of three phases. The project was implemented between 2017 and 2019 in nine counties where social-cultural practices accommodate this form of violence against women and girls (VAWG). The counties include Baringo, Embu, Garissa, Isiolo, Kajiado, Kilifi, Migori, Taita Taveta, and West Pokot. The project’s specific objectives varied by the counties covered and the phase of implementation. Among others, they included developing the capacity of community members to challenge FGM and promote girls’ education, strengthening women’s and girls’ networks to advocate for supportive of laws and policies, empowering girls with social and micro economic skills to challenge FGM, and advocate for duty bearers to implement laws and policies to eradicate FGM.

As the project drew to its close towards the end of 2019, AAIK called for an evaluation of three-year project. The objective was an independent assessment identifying the extent to which the project achieved relevant outcomes. RTA (Julius Nyangaga, Charles Nduhiu, Esther Kihoro, Grace Njoroge, Edwin Mugo, Emmaculate Kiptoo) carried out the evaluation using a combination of the OECD criteria (analysing relevance, efficiency, effectiveness, impact and sustainability) and outcome harvesting. The evaluation data was from document reviews, informant interviews focus group discussions and survey among beneficiaries. Cases studies of the most significant change were developed to demonstrate specific examples of observed or experienced transformation.

End term evaluation of WHH NIA and PACIDA projects (Feb to Sep 2018)

Welthungerhilfe (WHH) is one of the largest non-government aid agencies in Germany. The organization provides integrated aid: from rapid disaster aid to reconstruction and long-term development assistance in conjunction with local partner organizations. WHH Kenya has a country office in Nairobi, with intervention areas in mainly arid and semi-arid regions namely Makueni, Kitui, Kajiado, Narok, Tana River, Marsabit and Turkana. Other interventions areas in non-arid areas include Siaya, Kakamega, Bungoma and Vihiga County. The Neighbours Initiative Alliance (NIA), in partnership with WHH-K, implemented a project in the two counties, dubbed “Strengthening Resilience of Agro-pastoralist and Pastoralists in Kajiado and Narok Counties” between 2016 and 2020. The aim of the project was to strengthen the resilience of agro-/pastoral households and reduce their susceptibility to hunger and malnutrition. WHH-K also partnered with PACIDA in Kenya and AFD Ethiopia to implement a BMZ funded project “Enhancing resilience of agro pastoralists and pastoralists in the Borana zone in the Dillo, Dirre, Miyo, Moyale, and Teltele districts, Oromia Region, Ethiopia and in the North Horr, Maikona and Sololo sub-counties in the Marsabit County in Kenya”. The two organizations (NIA and PACIDA) commissioned RTA (Julius Nyangaga, Charles Nduhiu) for an external end-of-project evaluation to show how their interventions have led to long lasting change in beneficiary communities, and the effect of any other confounding factors.

The evaluation applied OECD/DAC criteria of relevance, effectiveness, efficiency, impact and sustainability as a guide to key evaluation questions. It was an in-depth analysis and documenting of qualitative and quantitative outputs and outcomes achieved in the various phases of implementation. The evaluation reports also presented challenges faced, lessons learnt and recommendations for the partners and relevant stakeholders for subsequent phases of the project.

Evaluation of CAP YEI 5 Year (2011 – 2016) Program (Mar 2016)

In 2015, Mastercard Foundation called for a final evaluation of CAP YEI’s (first) 5-year project of operation that was to end in 2016, with the findings constituting inputs towards a subsequent extension. This means that the evaluation was both summative in that it captures the achievements and lessons of the first project, and formative in that the findings were to help shape the next phases. Julius Nyangaga (from RTA) was part of the evaluation team from New Economy Development Group (NEDG) that carried out the evaluation. The overall evaluation approach was based on a utilization-focused evaluation (UFE) decision-making framework that emphasizes on ownership of the analytical design by primary evaluation users.

The project accomplishments are reported with emphasis on the short-term outcomes achieved. The findings focused on decreased spheres of influence taken from the Outcome Mapping tool that acknowledges that a project can only be held accountable for outputs and ‘expect-to-see’ outcomes. These elements can be measured in the short term (in this case during year 4 of a 5-year project). A number of ‘like-to-see’ outcomes that have already emerged were described, but there was no attempt to track changes beyond that point due to the short duration of the evaluation as well as the realization that impact depended on factors beyond the control of a project.

Final Evaluation of CORDS support to Innovative Operations Research Projects (Nov 2015)

In 1999, the Rockefeller Foundation advocated and supported the establishment of disease surveillance networks (DSNs) that operate across national borders’ for early detection and prevention of pandemics. The Foundation helped the creation of DSNs in regions of Asia and Africa. Among its early success was launching of CORDS (Connecting Organizations for Regional Disease Surveillance) to support and coordinate shared learning across the networks. CORDS’ mission was to support regional and sub-regional disease surveillance networks for sharing key information and real-time data, and organizing collaborative efforts to improve their capacity to respond to infectious diseases.

In late 2015, RTA (Julius Nyangaga) and Ziad Mousa (an evaluation consultant from Lebanon) were commissioned to carry out a final evaluation of the CORDS network. The assessment was to establish intended and unintended positive and negative internetwork change and data sharing did the project bring about, perceptions on how the current project was able to allow for impacts in the particular fields where they were working, and how/if the internetwork collaborations were critical aspects to make this happen. The evaluation specifically covered the period from which CORDS was launched in 2012 to the end of the period in 2015. This was a “Developmental Evaluation” which encouraged engaging the project management, partners, beneficiaries and other stakeholders in the evaluation. The evaluation was a “formative final evaluation”, whose results was to feed into future work by the programme. Following preliminary consultations between the evaluation Team, CORDS and the BMGF, the analysis focused on a set of questions that supported utilization interests yet fitted in the standard OECD criteria of effectiveness, efficiency, impact and sustainability. These areas of enquiries were used to develop the checklist used in final data collection. Information was collected from surveillance networks in Jordan, Albania and Tanzania.

The evaluation made conclusions and recommendations along the four main objectives of CORDS, i.e. (1) DNS formation and strengthening, (2) their capacity, development and management of innovative projects, and (3) their learning and sharing.

Final Term Review of Sikika 2011 – 2015 Strategy (Jun 2015)

Sikika Tanzania, a health advocacy non-governmental organization NGO, provides an oversight role in the performance of the health sector aiming at ensuring there is provision of quality health services to all Tanzanians. In contributing to the progressive realization of rights and capabilities in the health sector, Sikika’s 2011-2015 Strategic Plan aimed at increasing health sector budget efficiency, transparency and accountability at both central and local Government levels, financial allocation, equitable distribution of health workers and their adherence to professional ethics, and availability and accessibility to quality medicines and supplies at all levels of health care delivery.

In May 2015, Sikika called for a final term evaluation of its strategy. The evaluation was a case analysis of the organization’s strategic interventions social accountability monitoring (SAM), research/budget analysis and media advocacy campaign (MAC) and using them to assessing the program’s effectiveness, efficiency, relevance, and sustainability. The evaluation, undertaken by RTA (Julius Nyangaga, Charles Nduhiu, Elinasi Monga, Pauline Teveli) also described the outcomes that the organization achieved during the five years of operation.

The evaluation was participatory and strongly followed the conceptual framework of outcome mapping, but primed to harvest other outcomes related to the organization’s interventions. The data was both qualitative (especially from narratives and literature reviews) as well as quantitative (from records and surveys).

CCAFS’ Outcome Assessment (Mar 2015)

The CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) seeks to promote a food–secure world through the provision of science-based efforts that support sustainable agriculture and enhance livelihoods while adapting to climate change and conserving natural resources and environmental services.

Through the partnerships, CCAFS integrates thematic research (on adaptation, risk management and mitigation), facilitates user-driven research and enhances science-policy dialogues for up-scaled sharing of knowledge, tools and approaches. In 2012 through 2013, the regional program started the process of transitioning field-learning sites into “Climate Smart Villages” – where research partners and developing partners worked with farmers to test portfolios of promising climate change adaptation, risk management, and mitigation interventions. After 3 years of operation, the CCAFS East Africa Regional Program contracted Julius Nyangaga of RTA for an internal assessment of the impact the regional program has made and the outcomes achieved in Uganda, Kenya, Tanzania and Ethiopia during its operations. This assessment was to highlight the impacts the program has made in influencing behaviour change, examine the use of the knowledge and data generated by CCAFS and provide recommendations for future activities.

Evaluation of UNDP’s ATMS Project, Tanzania (Oct 2014)

The African Training and Management Services (ATMS) Tanzania project was started in 1989 by the International Finance Corporation (IFC), the United Nations Development Program (UNDP) Regional Bureau for Africa, and the African Development Bank (AfDB). Its aim was to support the management capacity of African companies so that they can participate in the global market competitively, profitably, and sustainably. The project has been implemented by the African Management Services Company (AMSCO BV). From 2006, AMSCO had placed 111 managers in 49 companies (or Beneficiary Organizations, BOs) under the ATMS project. This was with an exemption from taxes with the expectation that such relief would be recovered through increased corporate and employee taxation. In addition, the managers were tasked to build the capacity of the BO staff through training, coaching and mentoring.

AMSCO commissioned an external evaluation to analyse the impact the AMSCO Managers have had on beneficiary organisations between 2010 to 2014. RTA (Julius Nyangaga, Sarafina Nyaga, Elinasi Monga) was assigned the study based on 16 randomly selected BOs from the 49 BOs. The analysis was a comparison of key performance parameters between 2010 (as the Baseline) and the ‘End-line’, which was either 2014 or the end of the ATMS support (for ‘Completed Projects’ Where support for the BO through ATMS ended ). The sampled BOs showed a doubling in the number of positions created during the period under review (an increase of 96%). While this was as a result of some companies starting from scratch, most of the growth was a result of increased business profitability and recovery from loss-making to viability. A total of 3,183 staff members from the sampled companies went through in-house (internal) training with 45% being female staff. That was an average of 227 employees (with 112 of them or 46% being females) internally trained per company. A total of 87 staff members (37 of them or 43% being females) from the same companies were taken to special skills training where 37 were female. Eleven staff members, including 8 females were supported for academic courses. Following this, a total of 45 staff promotions were reported by the sampled BOs, with one upgraded to regional manager level, 13 to country level and 31 to departmental level.

The evaluation showed quite a significant growth in gross revenues earned by the sampled BO companies, which more than doubled their gross revenues from USD 10m to USD 20m.

Mid-term Evaluation of the CCAP Project, Tanzania (May, 2013)

The Climate Change, Agriculture and Poverty Alleviation initiative (CCAP was a partnership between five civil society organizations (CSOs) with specific experience in agriculture. The CSOs were ActionAid Tanzania, the Tanzania Organic Agriculture Movement (TOAM), and Tanzania Forest Conservation Group (TFCG) working with grass-root networks of farmers (MVIWATA) and rural communities engaged in participatory forest management (MJUMITA). The innovative partnership was to bridge the gap between NGOs traditionally focused on forest conservation and those working on agriculture. The ultimate goal of the CCAP Project was the reduction of poverty amongst small-scale farmers in Tanzania, and widespread adoption of climate resilient, low emission agricultural practices. The intermediate objective was to support the development and implementation of policies and strategies that prioritize support to small-scale farmers The project was in pilot sites located in six villages of two Districts: Kilosa (Lunenzi, Ibingu and Kisongwe Villages) and Chamwino (Mahama, Nzali and Manchali.)

In 2013, CCAP called for a mid-term evaluation of the program. RTA (Julius Nyangaga, Charles Nduhiu, Harrison Rware) was assigned to carry out the study whose evaluation interests/questions covered impact, effectiveness, efficiency, relevance, sustainability, and institutional capacity and risk assessment. Data collection took place in Dar es Salaam, Morogoro, Chamwino and Kilosa Districts, combining qualitative and quantitative data from interviews, focus group discussion and report extracts. The evaluation demonstrated how the project had covered the major steps of the Rapid Outcome Mapping Approach (ROMA). This entailed how the project had mapped the political context, identified key stakeholders and desired behavioural changes, developed intervention strategies, analysed and corrected, and developed monitoring and learning frameworks

End-of-Project Evaluation of two DFID-DDP projects (Oct, 2019)

In 2019, Christian aid made a call for a final evaluation of two projects implemented in Marsabit County (Kenya) between 2017 and 2019, and funded by DFID’s Deepening Democracy Programme (DDP). The first project The Accountable and Inclusive Management of Natural Resources and Livestock Production Systems in Marsabit County”, was implemented by Concern Worldwide. The second project Strengthening Citizens-County Government Collaboration in Water Governance for Enhanced Community Resilience in Marsabit County” was implemented by Christian Aid/MIO-Net.

The main objective of the final evaluation carried out May to October 2019 by RTA (Julius Nyangaga, Harrison Rware and Charles Nduhiu) was to present an independent assessment of the respective projects, analysing the extent to which they achieved the expected results, key outcomes and the drawing of relevant lessons for the implementers and partners. The study adopted Outcome Harvesting (OH) approach to the evaluation objectives.