Measuring the impact of our 2014-2016 programme

Every three to five years, Rikolto assesses the impact of its programmes, to check whether our interventions have the desired effect, and to learn where we fail and need to adapt our approach. These impact assessments consist of an intensive process of data collection and interviews with members of the farmers’ organisations and different stakeholders.

As 2016 came to a close, so did a three-year global programme funded by the Government of Belgium – the perfect moment for Rikolto to measure the impact of its work across the 12 countries covered by this programme. (Our activities in Vietnam and Mali are funded with other means, just as other programmes that our regional offices implement, which were not subject of the 2016 impact assessment).

The main question burning on our lips: did we manage to bring about the positive change that we had hoped to set in motion?

The objective of Rikolto (VECO at that time) in the 2014-2016 programme funded by the Directorate General for Development of the Belgian Ministry for Foreign Affairs, was to render agricultural value chains sustainable and inclusive of smallholder farmers. This way, we aspired to improve the lives of over 100.000 farmers in 12 countries in Africa, Asia and Latin America. However, working with all those farmers individually would have been rather impractical (how to reach all those farmers?), inefficient (how many people and how much money would we need?) and unsustainable (who would ensure the continuity of our work?).

That is why Rikolto accompanied farmer organisations to become reliable business partners in the value chain and effective service providers for their members. In so doing, we give smallholders a stronger voice and better bargaining positions, turning them into bigger players in their societies. We also supported sustainable and inclusive commercial relations between the different actors in the value chain (such as input suppliers, service providers, buyers and retailers). These activities were expected to lead to inclusive business models for small-scale farming that would inspire other companies, farmer organisations, service providers, governmental and non-governmental actors, to implement these models at a scale beyond Rikolto’s reach.

So far, so good: there is a logic to this strategy and it sounds reasonable. Yet, how do you measure the impact of this in a way that allows you to make a connection between the changes you come to observe and Rikolto’s work?

In several cases, we are not the only actor working with a farmer organisation; oftentimes, the ministry of agriculture would have a local representative offering technical assistance, or other non-governmental actors supporting the organisation in a different area of its work. What a farmer organisation ends up doing, will be the result of the skills and attitudes of its staff, influenced by the combined efforts of all other actors engaging with them, and by the opportunities to carry out their jobs.

When you go to the next step down the effect chain, it is even harder to establish why and how the lives of farmers have improved (or deteriorated). Rikolto supports farmers working in a specific value chain, e.g. cocoa. Yet, a cocoa farmer may also cultivate plantain, herd cattle, have a side-business selling organic manure, provide transport services and receive remittances from family members, to name but a few of the factors that influence a farmer’s ability to improve the livelihood of herself and her family. So do we only measure the income difference derived from a very specific chain? Or do we look at his/her family business as a whole? Long story short: measuring the impact of our work is a challenging endeavour. Fortunately, we are not the only ones in the game. There are several other organisations and research institutes who, like Rikolto, continuously strive to improve the tools they use to measure impact in complex environments.

For the 2016 impact assessments, we have updated our measurement frameworks to better reflect our intervention strategies and the changes we sought to realise. This revision used inputs from recent academic research on assessing the impact of capacity development of farmer organisations and from the recommendations of a study on impact research methods for NGOs carried out by the Belgian Government’s Special Evaluation Service.

The resulting framework proposes a mix of methods, both qualitative and quantitative, combining primary (farmer surveys) and secondary data (Rikolto’s monitoring reports) with the triangulation of findings through focus group discussions and interviews, to establish the effectiveness and relevance of Rikolto interventions.

In the context of our work, impact touches on the following domains:

• Improved livelihoods of smallholder farmers (m/f);
• Improved business and organisational capacities of targeted farmer organisations;
• Improved institutional environment (public/private policies, regulations, laws, etc.) for inclusive and sustainable sub-sectors and inclusive modern markets.

In relation to these domains, the impact assessment framework poses three fundamental questions:

1. Which changes do you observe at outcome level and to which extent has Rikolto’s intervention contributed to these changes? Rikolto works with partners (farmer organisations, buyers, service providers) to strengthen their skills to make agricultural value chains more sustainable and inclusive of smallholders. Have the capacities of these partners changed and what was the role of Rikolto therein? In other words, were our interventions effective in bringing about the change they aspired to realise?

2. Which changes do you observe at impact level and to which extent have Rikolto’s interventions contributed to these changes? As Rikolto’s partners become more skilled in promoting sustainable and inclusive value chains, this should translate into better working conditions for smallholders in those value chains (e.g. better prices, increased yields, clearer buying arrangements), leading to better lives for themselves and their families. Have farmers’ livelihoods indeed improved and how much? Did these improvements have some connection to Rikolto’s work (through its direct partners, that is) and do these improvements matter in their lives? In answering this question, we learn what the impact of our work has been and its relevance for the targeted farmers.

3. Which lessons can you draw from the effectiveness and relevance analyses of Rikolto’s interventions, in view of harnessing the impact of future interventions in related areas? By carrying out the analyses to answer the first two questions, we also learn a lot about how efficient Rikolto has been with its resources and how sustainable its interventions have been. Did we put our human and financial means to the best possible use in view of attaining the desired impact? Did we contribute sufficiently to an environment in which the realised changes will have continuous impact in the medium and long run? How else can we improve our work as an organisation and the strategies we put in place?

There were three big steps in our 2016 impact assessment process:

Collecting data on changes at outcome and impact level

Three main sources provided information on changes in relation to Rikolto’s work:

• Farmer surveys offer us data from a random selection of targeted farmers on income, profit margins, diversification of income, environmental production practices, gender relations and the role of youth in the value chain.
• SCOPEinsight assessments rate the professionalisation of farmer organisations on several dimensions, ranging from internal and financial management skills to business-savviness, access to finance and their influence on trading relations (Rikolto started to use this tool in 2016 – a conversion table allowed for comparisons with the matrix used to calculate the baseline and intermediate scores).
• Half-yearly monitoring reports, in which colleagues update the outcome indicators, capture evolutions in sector policies, and collect stories that illustrate the dynamics behind the observed changes.

Carrying out a contribution/attribution analysis

A critical step in the impact study is to connect the implementation of Rikolto’s interventions with the observed changes at outcome and impact level. To ensure the validity of these connections, a set of interviews and focus group discussions were organised with partners, farmers and key stakeholders.

These discussions also helped to distinguish between those observed changes at outcome and impact level that are attributable to our interventions (i.e. the observed changes have been brought about uniquely thanks to the intervention and would not have occurred without Rikolto) and those to which our interventions have contributed (i.e. the intervention played a role in bringing about the observed changes but Rikolto was not the sole reason why they occurred).

Finally, a counterfactual reflection complemented this analysis: what would have happened if Rikolto had not implemented its interventions? In absence of comparison groups, it was not possible to carry out a profound counterfactual analysis during the 2016 impact assessments. Starting from 2017, however, Rikolto shall pilot the use of comparison groups in some interventions to allow for increased rigour in its assessments and gain an even more thorough understanding of how it brings about change.

Disseminating the results and collective learning

Upon terminating the impact assessment studies, Rikolto shares its results and recommendations with its partners organisations, key stakeholders, donors and the wider public. Furthermore, these studies nurture the internal discussions and learning processes in the organisation as a whole and between the regional offices. We keep track of good practices, lessons learnt and challenges, and exchange them among the topical implementers within our offices, to improve our intervention strategies and attain a greater impact.

You can consult the 2016 Rikolto (VECO at that time) external impact assessments by clicking on the following links. The 2016 internal impact assessments are available upon request.