Impact Evaluation

Expert opinion

An overview of the area - what do we mean by impact evaluation?

Impact evaluation is associated with jargon and acronyms, RCTs, counterfactuals, SROI and so on, that can make it seem complicated.  At a very basic level impact evaluation is about asking the following two questions:

•       what difference are we making? and,

•       what is making the difference?

But these questions are not as simple as they look. An impact evaluation explores what has changed (the outcome) and whether that change is attributable to the particular project or intervention being evaluated. The challenging part is being able to isolate the impact and demonstrate that it is due to whatever is being evaluated, rather than due to something else in the environment or peoples’ lives. Attributing change in this way is not always possible.

There are different methods for generating evidence of impact and conventional ‘standards’ of evidence encourage using a method which leads to higher confidence and lower of bias in results. The table below shows how different evaluation designs provide higher or lower confidence and bias in results. A wide range of practical considerations will influence the final approach taken, and you may want to seek some support to help you work them through.

 

To maximise the opportunity to measure impact evaluation must be planned from the outset of the project or intervention, otherwise you may miss important opportunities for data collection and limit the methodologies that can be employed.

Next steps and practical examples

When commissioning an evaluation these are some questions to ask before starting as they will provide a better understanding of what is needed and what type of evaluation might be possible.

1.       What stage of development is your project or intervention at (e.g. an innovative new idea or a well established model, or somewhere in between)?

2.       What do you want to know, for what purpose, for what audiences?

3.       When do you need to know this by?

4.       What evidence already exists that is relevant to what you are doing?

5.       What data do you currently have or are you likely to have in the near future?

6.       What resources do you have including money, time and expertise?

7.       Who else wants to know about this? Can you work together? Would your objectives be properly aligned or would it over-complicate things?

8.       What can you do yourself and what external skills and support do you need to access?

An important starting point for designing an impact evaluation is a clear understanding and description of the project or intervention to be evaluated. A theory of change describes how resources (inputs) are used for different activities (outputs) and the difference these activities make for the people or organisations involved (outcomes), and whether these differences would have occurred anyway (impact). A theory of change is therefore a useful way to identify what should be measured. Putting together a theory of change is also useful for surfacing and sharing tacit knowledge and implicit assumptions about how things are working and to aid strategic and operational planning.

For example, for one charity the conversations they had during their first theory of change discussion surfaced how staff had very different ideas about how one part of the programme would work.  As a result they were able to reach a new, shared understanding before investing resources in starting to implement their ideas. They also produced a one page visual during the session; it captured how their work was expected to lead to a range of inter-related outcomes for young people. This proved to be a useful and effective addition for use in funding applications, even before the evaluation had been started.

The methods used to evaluate impact will be dependent on whether certain conditions are met. Circumstances such as what data and resources will be available and the timeframe for the evaluation will influence what is possible. A number of factors are necessary to attribute impact to the project or intervention being evaluated:

o   the way a project or intervention  works will need to be reasonably well-defined and unlikely to change during the evaluation;

o   it should  be possible to manage the implementation and delivery of the project or intervention to make sure it does not change in a way that would fundamentally disrupt the evaluation;

o   it should be possible to select a group of people using  the project or intervention to take part in the evaluation, and a separate group of people who are not involved in the project but who are similar in other respects (a control group); and

o   the control group should not be using anything similar to the project or intervention being evaluated.

 

Process and economic evaluation are also important to consider alongside an impact evaluation, if there is a desire to understand how a project or intervention is working (for example to replicate it elsewhere) and the economic implications (for example if it is necessary to put together a business case for future funding or investment).

The evaluation process and eventual findings, cannot replace the hard work of thinking deeply about what you do, the difference that you are trying to make and how to communicate this. Analysis of data and the ‘final report’ are often not enough to create change. Findings must also be interpreted or translated to show the practical implications for different audiences and stakeholder groups.

Outlook for future development

Impact evaluation becomes increasingly important in a cash strapped environment and when different models of funding proliferate.  Impact investing, social impact bonds, payment by results, commissioning for social value all require methods for understanding the actual and relative impact of different projects and interventions.

Alongside traditional methods to determine impact there is an important role for methods that empower research participants and get closer to the voice and experience of people involved in the projects and interventions being evaluated.  Peer led research, where people are trained and supported to capture data from those around them is one example of this. Ethnography, blogging, online forums, video diaries and service design techniques such as customer journey mapping, are just some of the other tools and methods that are available to complement impact analysis and provide rich contextual understanding to support findings.

Resources

Theory of Change community and resources (some free and some fee-charging)

http://www.theoryofchange.org/

Anderson A.A (2010). The Community Builder’s Approach to Theory of Change: a practical guide to theory development. (New York: Aspen Institute).

http://www.aspeninstitute.org/sites/default/files/content/docs/rcc/rcccommbuildersapproach.pdf

Clappison A (2011) ‘Theories of change: an expanding resource list’ Research to Action 11 May 2011, accessed16 August 2012

http://www.researchtoaction.org/2011/05/theory-of-change-useful-resources/

http://www.civilservice.gov.uk/wp-content/uploads/2011/09/the_complete_magenta_book_2007_edition2.pdf

Khandker, Shahidur R.; Koolwal, Gayatri B.; Samad, Hussain A.; (2009) Handbook on impact evaluation : quantitative methods and practices World Bank

http://econ.worldbank.org/external/default/main?pagePK=64165259&theSitePK=469382&piPK=64165421&menuPK=64166093&entityID=000333037_20091210014322

Glossary of common terms in impact evaluation

http://www.3ieimpact.org/media/filer/2012/07/11/impact_evaluation_glossary_-_july_2012_3.pdf

Social Impact Analysts Association mentoring scheme

 http://www.siaassociation.org/resources/siaa-mentoring-scheme/

This expert opinion is tagged under:

  • Impact measurement