Impact Science

Make the most of your research by gathering evidence of the impact

Learn how to effectively gather evidence for the impact your research created and reap its benefits

Dr Iain Coleman, who leads the work of Impact Science in the UK, joined us to participate in a series of insightful conversations. Dr Coleman provides comprehensive impact and engagement support to universities and researchers, in the context of the REF, the KEF and the broader research impact and public engagement agenda.

This week’s blog is based on Dr Coleman’s commentary on How and Why to Gather Evidence and Ensure Exhaustive Areas of Impact.

What do we mean by evidence of impact?

We mean any kind of hard evidence that shows how research in a university has benefited the wider world, such as changes to public policy, changes to healthcare practice or commercialisation of a product.

We don’t mean evidence of process. The fact that you talked to a lot of people, or your paper was downloaded a lot, is not evidence of impact. A classic example is when people are asked for evidence of impact and say “I’ve given lots of talks” or “I’ve been on television”. These activities might lead to impact, but you need evidence that some change has happened as a result.

Why is it so important to gather evidence of impact?

In the UK context, the REF demands that universities present evidence of their impact and allocates funding accordingly.

But it goes much further. If you’re going to change the world through your research, then it’s important to understand the real effects of your work. So it’s vital to gather evidence rather than relying on your intuitions. Richard Feynman, the Nobel prize winning physicist, said that “people are easy to fool, and the easiest person of all to fool is yourself.” Evaluating impact is about not fooling yourself. It’s about having objective reasons for believing you are doing well – or finding out you’re not doing well, and changing course.

What are the steps and stages of evidence gathering?

There’s no one size fits all answer, but in general it’s a process of identifying the likely beneficiaries research, the stakeholders that you’re already working with, who else might find your research to be of value, and mapping that out.

Sometimes that can be a desk exercise – if you have a clear idea of who you’ve been working with and who’s likely to be benefiting from your work – or it might be a process where you identify one or two beneficiaries and ask them about other people who have taken up your research.

For example, you may have been working with a hospital to develop some medical technology. That hospital is your primary beneficiary: you would ask them how they and their patients have benefited, and then go on to ask if other hospitals have taken up this technology. This can become a snowball effect, where you start with one or two beneficiaries and build up from there.

Having identified beneficiaries, you have to get evidence from them. That might simply be a testimonial, or it can be other data such as sales figures or audience numbers. The key point is to engage with the relevant stakeholders, and understand what evidence they have already gathered and can provide to you.

You can also do a more formal evaluation: constructing a theory of change, setting out how you believe the research has had impact, and then undertaking more rigorous evidence gathering. This may involve reaching out to stakeholders with structured interviews or questionnaires, or doing in-depth desk-based research, before analysing the quantitative and qualitative data. This kind of analysis can be invaluable for understanding more complex and far-reaching impacts.

What are the best practices that universities should follow?

Engage with stakeholders at an early stage, to make sure they understand your need for evidence. It can be awkward if you’ve been working with someone for a few years and then you suddenly ask them to provide impact evidence. This can put the relationship in a different light. It’s better to say at the start, “We’re going to work with you, and as part of that we’re going to need a testimonial in due course.” It’s also important to build in evaluation processes from the start, especially when your impact involves public engagement or public understanding. Often you are looking for evidence that people’s understanding or opinions have changed as a result of your event or outreach. You can gather that information at the time. You can find out what people thought or felt beforehand, and then see how that has changed afterwards by means of questionnaires, email follow-ups, even focus groups. But if you come in after the event, obtaining this evidence can be impossible.