Social impact measurement: a challenge


Magazine article
Charities are increasingly forced to face the harsh reality that funding streams previously open to them, particularly local and national government funding, are not likely to be renewed. For many charities this shrinking pot has been a real wake-up call, forcing them to look elsewhere for funding support. Not only are many at a crossroads in terms of deciding how they will continue funding the delivery of services, but the thought of applying for funds through previously unexplored channels can be daunting. This is because it gives rise to a fresh level of scrutiny which, although it should be welcomed in a sector seeking to increase its transparency and accountability, is often anything but.
It seems logical that charities will turn towards private funding as other sources are withdrawn. Arguably this creates an opportunity for trusts and foundations to assume a position of responsibility within a sector facing declining public confidence by demanding the bar be raised where standards of governance, achievement of outcomes and demonstration of impact are concerned.
There are many attractions to an increase in philanthropic funding, not least because it tends to be less bureaucratic, with application processes entailing fewer administrative loopholes. It can often be more flexibly used than other types of funding and can offer a more supportive relationship between funder and recipient. Philanthropists are often better placed to take risks to obtain greater social return and promote innovation. There needs to be room for experimental work to find new solutions to the social and other challenges we face.
Arguably this is where true philanthropic funding should operate at its best, i.e. where really innovative ideas merit funding but are not sufficiently tried and tested to generate evidence of impact required by state or other funders. Philanthropic funders may be more inclined to offer non-financial support alongside funding, take a more flexible approach as to how results are achieved and be prepared to push boundaries. Furthermore they should be more prepared to fund outcomes that are less evident in the short term – such projects being less attractive to funders who are under pressure to stakeholders to justify their decision.
That said, in a climate of rapid decline not only in funding but public confidence in charities, it is more important than ever to focus resources on demonstrating – as opposed to merely claiming – impact. As the demand for funding from trusts and foundations rises, are there ways in which they can promote the value of demonstrating impact amongst the sector and address the challenges associated with impact measurement?
We think that funders have a valuable contribution to make in helping recipients demonstrate the impact their investment has achieved. This starts with requiring recipients to record data for impact measurement, and funding them to do so. This can be done by undertaking an analysis of outcomes to demonstrate the impact of using that data to create a body of knowledge that can be shared amongst, and used by, the relevant parts of the sector.
It can be the case that charities will not record data unless specifically required to do so by a particular funder, often in relation to a particular project, and then they stop upon completion of the project. Some recipients perceive impact measurement as a boxticking exercise to satisfy their funder’s stakeholders. They do not recognise the wider value of the data they record. Of course there are examples of charities who absolutely do see the wider value in such data. One such example is Community Links, a charity based in Newham, East London with over 30 years of experience working with local people to support children, young people, adults and families. On a national level, the charity shares lessons with government and community groups across the country to achieve social change. Community Links recognises the need to develop measurements of long-term impact as a way to bring about sustainable change. Evaluation1 of one of its projects showed not only direct impact from its work, but also a finding that 80% of local people interviewed felt that the areas in which that work had been carried out had changed for the better in the 12 months since it started. This has implications for measurement of impact and Community Links has sought to map the full range of outcomes – categorised as (i) primary (which is essentially quantitative and conventionally the only one recognised and paid for by government), (ii) aggregated (long-term outcomes from collaborating with other partners) and (iii) rising tide (sustained and collective) – in order to understand the full potential impact.
Community Links recognises that primary outcomes ‘have, at best, only a partial relationship to the changes that the funder wants to achieve’. They see opportunities to develop a more collaborative approach which ‘gives proper weight to the three categories of outcome’ and produces interventions which make the most efficient and sustainable use of resources and consequently deliver a higher return on investment. This faith in the value of impact measurement is not sector-wide. One key underlying issue is the lack of an agreed, sector wide definition of social impact. If the sector is serious about demonstrating impact then surely a more unified approach is required as to the meaning of impact in its various contexts.
It is not as simple as the sector lacking the necessary skills to successfully measure social impact (indeed it has taken other sectors a long time to develop the techniques required) but more that there is a need to demystify how impact should be measured. The act of measuring social impact can be incredibly complex (after all, it concerns the effect of an action or intervention on many individuals) and there is no consistent system of metrics used across the sector. It is also easier to measure impact in some areas than others – education, for example, has standardised measures in place which enable changes to be measured over time.
Other areas do not have such measures, making it more challenging to assess outcomes, and certain types of impact are not easily quantifiable or may not accrue until years after the intervention has occurred. In the context of providing free legal advice, for example, if you were evaluating the impact of facilitating a fair financial settlement in a divorce claim, a lot of the associated benefits would accrue over an extended period of time. The consequences could range from achieving amicability between parties, the mediumand long-term benefits on children, minimising the potentially destructive effect of acrimonious divorce on the family unit and the associated ripple effect that this can have on physical and mental health and wellbeing, the ability to remain in employment, reducing dependency on the state and sustaining successful future relationships – the list goes on. It is, of course, impossible to capture all of this straight away.
Furthermore, the variety of methodologies for measuring impact used across the sector, combined with a lack of consistency amongst funders in terms of what data they require and how it should be presented are all points of confusion for grant applicants. Because methodologies follow trends and cycles which differ according to who the funder is, it is vital for recipients of funding to be clear as to who their audience is when evaluating impact. A charity may produce detailed evidence demonstrating impact in a certain area but it depends on the funder as to whether it will be relevant or not, since one funder’s interpretation of data may differ from that of another. The complexity involved and the consequent lack of understanding within some charities as to what the different methodologies required of them mean, exacerbated by a lack of familiarity with the jargon used, can be off-putting. This lack of consistency also gives rise to a question mark within some charities as to the value of the data.
An example of a current government-favoured methodology is the randomised control trial (RCT) which seeks to prove causality between intervention and outcome, but for some charities the prospect of this is intimidating. Aside from allocating resources to facilitate an understanding within a charity of what this would actually entail, the very concept of an RCT does not sit well with some charities given that inevitably it will require a control group – and hence a proportion of people will not receive intervention. Some charities struggle with the premise that, in order to get funding, they will have to turn some people away. Clearly for some charities – a rape crisis centre, for example – it would simply not be appropriate to implement an RCT.
Some charities are disinclined to measure impact due to an ethical conflict arising from the prospect of diverting resources away from the act of carrying out their objectives. For others it is a resource issue: even where charities have an appreciation of the value of impact measurement, they lack the time, money and people necessary to develop an understanding of the evidence-based methodology required of them by particular funders and to collate and present the data needed in the requisite form. Ultimately this can prohibit smaller organisations accessing funding from those funders who require their recipients to demonstrate impact through complex methodologies and yet are not inclined to build into their funding the cost of enabling impact measurement to be carried out by the recipient.
It is worth noting that evidence from projects that have failed to achieve the intended outcome can itself be of significant value in indicating where improvements or changes need to be made and can feed into future delivery to achieve even greater success. However, charities are, for obvious reasons, risk averse by nature and with that comes the fear that taking a chance on something novel and untested could have a damaging effect on their reputation, ultimately impacting negatively on their relationship with donors. Hence there is need for funders willing to take risks and be clear in their support for charities willing to try out innovative approaches.
The sector needs to increase its efforts to educate charities in both the meaning and value of impact measurement; as well as to help charities understand the various tools and techniques used to measure it. Strides are being made to develop online tools to assist with the impact measurement process. Big Society Capital’s Outcomes Matrix is a good example.
Many charities lack awareness of resources to help them measure their impact. Big data is one such resource and it clearly offers huge potential for the sector to identify and target areas of need. The voluntary sector is playing catch-up with other sectors which have used big data successfully, although efforts are being made to make more use of existing data to assist with impact measurement and to tap into other sectors’ resources. Analysis of big data has been used in a public health context, for example using Twitter data to predict outbreaks of flu, but how helpful it is will vary according to the aims of the particular organisation.
A big issue for the sector is whether or not the required datasets exist; and if they do, whether access is permitted. A ten-year initiative launched in 2012 called Inspiring Impact has created ‘The Hub’, which has collated a variety of datasets – The Justice Data Lab, The Centre for Social Impact Bonds Toolkit, Homelessness Statistics and Crime Statistics being just a few examples – to facilitate access to data by charities and social enterprises in order to help them better assess their impact. More investment is required, however, to improve the quality of generation, retention and access to data. There is no doubt that there is more trusts and foundations could – and in our view, should – do to stimulate the production of datasets, to lobby for improved access to administrative data and generate more custom-generated datasets to identify trends, the potential for collaboration and areas of duplication. A good example of this is ‘Where the green grants went’, a series of publications coordinated by the
Environmental Funders Network.
There is no doubt that the government together with a number of sector organisations has been working to promote the message that charities need not be intimidated by impact measurement requirements. Inspiring Impact is managed by NPC and involves a number of sector bodies and impact measurement experts which aims to change the way the voluntary sector thinks about impact and make high quality impact measurement the norm for charities and social enterprises by 2022. Key to the programme is addressing the following questions:
  1. What does good impact measurement look like?
  2. How do we know what we need to measure?
  3. How do we measure it?
  4. How can we compare with and learn from others?
  5. What is the role for funders?
We believe that funders, in particular trusts and foundations, have a comprehensive and critical role to play in addressing the first four questions. This includes helping to demystify impact measurement by investing in the development of an across-the-board understanding of what is meant by social impact in its varying contexts, and developing techniques and tools for measurement and best practice methodological approaches. Funders themselves should adopt consistent methodologies which achieve a balance between a set of agreed high level values without leaning too heavily towards excessively detailed requirements that risk creating a barrier to funding. They also have a valuable role to play in educating the sector to better understand the value of impact measurement; encouraging a wider use of existing data, including lobbying for greater access to big data; collaborating to create more custom-generated datasets; and in seeking to improve habitual practice of, and improve platforms for, the sharing of intelligence and expertise within the sector. Crucial to achieving this is that they must be prepared to build into funding the resources required to enable recipients to record vital data in a way which is useful for the sector as a whole.
Philanthropic funders are in a unique position to help prevent the erosion of faith in social impact measurement, in particular by encouraging a move away from the damaging perception that it is something to be done simply to access funding and satisfy funders’ accountability. They can also encourage a greater commitment to the value of measuring social impact for the benefit of the sector as a whole by developing an appreciation of the value of data and its potential for improving outcomes in the long-term. We hope they take up the challenge.
This article was first appeared in Issue 11 of Philanthropy Impact Magazine. Click here to download the article in PDF.


This article is tagged under:

  • Impact measurement