Measure for measure


Magazine article

Before I became a trustee for Children in Crisis (CIC), I needed to see the impact of the funds. Coming from a financial background and my experience in the City, I am used to expecting evidence of a good return on investment. There are a number of charitable causes demonstrating impact; however, what does a charity do when its impact does not tick all the boxes?

Funders and donors have specific requirements when it comes to funding projects. Funding decisions can often take place in boardrooms, which can be remote places, far from the reality of the field. The evaluation on a return on investment (ROI) may be based on quantity, time taken to fulfil outputs, service delivery and some level of comparison. Measuring service delivery in poor communities around the world is complex and multidimensional and we would caution NGOs and charities not to take a simplistic approach to measurement. When quantitative and qualitative indicators are used for cross project comparison, there is a real risk that the indicators can provide misleading results and again we could urge caution in overall measurement and analysis.

Children in Crisis educate children in post-conflict countries such as Afghanistan, Burundi, DR Congo, Liberia and Sierra Leone. They work in the regions other charities and NGOs are not usually found (they were one of the first charities to work in Afghanistan, after conflict). They build schools and work directly with communities to enhance the lives of the children they educate, along with their families. Before building a school, CIC works directly with communities and engages them in every aspect of their project. I spoke to Koy Thomson, Chief Executive, about this. “Often donors do not wish to fund the long process of listening to people, nudging, negotiating, ensuring that everyone is included and no-one dominates. But it is the key to long-lasting impact. Prevailing culture and perceptions within the community can sometimes be seen as a stumbling block to community development. CIC see this as a learning opportunity,” says Thomson. “We aim to help people to improve what they already do rather than tear it down. If people get better at planning, working together and negotiating with outsiders, then they will use their time and resources much better, and may even secure additional resources.” The process is costly for CIC, but as Thomson says, “you could deliver a project with apparently fantastic returns on investment – if that was all you cared about, but if you have not improved the capability and effectiveness with which people solve their own problems, what really have you achieved of lasting value?”

ROI for an educational development charity working in very remote or conflict-affected communities, such as CIC, would require a more sophisticated economic model to calculate. “The world used to believe that the greatest ROIs in education come from picking the low hanging fruits,” says Thomson, “the easy places to work and access. It is certainly cheaper to educate a child whose school is by the side of a well-kept road within a functioning city in a peaceful country. But the fascinating thing is $10 spent on a child who has no access to services and faces great risks generates far higher knock-on benefits, for example in health, child protection and nutrition.” It is deep within CIC’s values to work in more challenging places and with the most vulnerable children. “If I had the money to invest in evaluating all of these knock-on benefits, I would,” says Thomson. “But I fear that it would cost far more than the educational programme itself.”

One scheme that the charity has set up to educate families and to tackle poverty, is through a Project Pamoja, a Savings and Loans Scheme for villagers in the Plateau region of DR Congo, set up in 2014. Its objective is to create, train and support groups (mainly female) on money management while reinforcing gender equality and social cohesion. The approach involved savings, micro-credits and a social fund (a mini-insurance for emergencies). The project reached 705 members, of whom 70% were women, in 30 Village Savings and Loan Schemes. The groups were supported by three field agents over three target regions. The total amount saved by the 705 members over the course of this first cycle was $17,909, which equates to an average of $25.40 per member. Following the repayment of all the loans, the total amount in the cash boxes was $26,332. This signifies a total profit of $8,424, or an average of $11.95 per member. This represents a 47% yield on savings. These results are tangible, but the intangible effects meant far more to the villagers than the finances. The top two reasons for borrowing money were health care and education; food came third.

Out of 142 Village Savings and Loan Scheme members, 69% took out credits to cover school fees. 78% say that they are better able to manage their money, which means they can plan better for future educational costs.

Amy Parker, the Programme Manager in DR Congo, carried out two levels of evaluation for Project Pamojo. She did an evaluation based on the evidence and impact based on a funding requirement, but she also went directly to the people in the village, to hear their views on the impact of the project. When speaking directly to the villagers, the impact was clear, but not quantifiable. People spoke of hope for building peace; of self-respect and respect for other ethnic communities. This is where the true value and impact of a project lies. Once again, there is a strong element of social development coming through. Improved social cohesion, community solidarity, peaceful conflict resolution, social reintegration and mutual respect all feature as positive changes seen since the implementation of Pamoja. “There is a lot of love and solidarity in our group. And this is despite the fact that we are not all from the same community,” said Albert Shoshi, Gitigarwa, Village Savings and Loan Scheme member, part of Project Pamojo. Mrs Nyamasomo Muhanga, another member said, “I have contributed financially to the building of our house. My husband now has a greater respect for me – now everything is managed transparently between me and him.”

The impact in these communities – social interaction (peace), security of family, a full understanding of identity and where individuals fit in society, equality and respect; all go back to our basic needs.

As a trustee of CIC, I am fully aware that there are more costs incurred to educate a child in a remote area, than in urban areas. Building a school in a town is less taxing than building a school in a remote village. In a town, however basic the infrastructure, there are systems in place to surround and support the school. Quality is harder to achieve in remoter areas. It is not surprising that qualified teachers are less likely to move themselves or their families to conflict zones. In remote areas teachers need to be recruited and trained from scratch and not just recruited. It costs significantly more to achieve the same results for educating a child in a conflict zone, than in a stable region. In the remote and unstable regions, you are more likely to find less access to education. In these regions the education supply and support from the state and from other NGOs is the poorest. In these regions, funds from philanthropy are needed the most. Naive impact indicators can at worst drive NGOs to work in safe, easily accessible areas where education provision is already adequate.

It is difficult to quantify levels of remoteness or to put a measure on political stability or levels of insecurity. It is even hard to work out how to measure the cost effectiveness of service delivery in these areas. This is not to say that impact cannot be measured. It is merely to argue for more sophistication and flexibility in the measures used.

Trustees and funders are usually looking for value for money, says Koy Thomson, so “we look to be effective, but not extravagant”. “However, if communities put such a high value on respect, wellbeing or peace, you must listen to them. I have learned much from the state of the art debates on ROI, but never have I been more humbled than hearing from the villagers themselves that the greatest value of our village loans schemes to communities was not the very positive quantifiable profits and economic exchange, but peace and inter-community relationship building that came as a by-product of people of different tribal and social backgrounds meeting to work together for the common good.”

This article first appeared in Philanthropy Imnpact Magazine issue 13. Download this article as a PDF.

This article is tagged under:

  • Impact measurement