Impact beyond the REF. Consider that for a moment. What does that mean? What would that look like? And with REF so often synonymised with impact in the UK, how can we extract ourselves from its dominant voice?
I was invited to deliver the opening plenary – alongside Melanie Knetsch (ESRC’s Strategic Lead for Interdisciplinarity, Innovation and Impact) and Professor Peter Kemp (Vice-Dean for Academic Affairs and Professor of Public Policy, Blavatnik School of Government, and University of Oxford Social Sciences Division Strategic Impact Lead) – to consider impact beyond the REF. It’s an odd concept because – let’s face it – REF isn’t going anywhere, so establishing ‘beyond’ is immediately difficult in tone.
Ultimately there are numerous ways to look at this. As my plenary colleagues considered the bigger strategic picture and the policy context, I chose to think about impact on-the-ground and in the lives of researchers, research managers and ‘users’ (more on that later). Here are my headlines:
First things first, impact existed before the REF. I hope you were sitting down
I don’t want to startle you, but REF didn’t invent impact. I know – it’s a shock. Of course we had impact before REF, we just didn’t really call it that. I started my research career in 2003 and anything beyond the academic world was generally captured by words like ‘applied’, ‘knowledge transfer’ or in my own discipline ‘(health psychology) ‘implementation’. Impact (with a big ‘I’) didn’t really hit our radar until its introduction to REF in 2011, and even then, only those involved in a REF 2014 submission had to invest effort in tracking, evidence gathering and causative narration. But impact is neither exclusive to or borne from REF. Please breathe again.
REF is a double edged sword (or ‘let’s elevate the value of applied work…..then make them dance for money’)
Unlike previous research assessment exercises, REF elevated the importance of research-driven real world change. For many of us, that recognition offered a welcome rebalancing of what it meant to ‘be an academic’. However, formalising impact within assessment agendas also introduced weighting, ranking and a financially incentivised need to drive bigger, stronger, more important and further-reaching effects. Whilst this has triggered investment and innovation, we’ve also witnessed a culture of anxiety and substantive institutional restructuring. If we can continue to use REF to catapult creativity, balanced with fair expectations (and reduced worry about repercussions of not doing impact), perhaps we can really deliver social benefit.
A need for impact literacy across the research system.
Impact is important, but not all research should have impact. We need discovery science, just as we need philosophical shake-ups of academic assumptions. But we also have a duty of care to pass it forward, whether that’s beyond the academic wall, or to other, perhaps more applied academic counterparts. Everything we know about impact says it can’t be templated and is achieved through multiple routes. Some are scripted, some are opportunistic, but all involve decisions about the best and most purposeful approaches. With so many possible permutations, it’s incumbent on the research community to become impact literate (paper here or OA version here). Let’s be clear, I don’t mean ‘more training for academics’. Of course training is needed, but what I’m talking about is an understanding of how impact ‘works’ across academics, research managers, institutional leadership, funders, publishers……..a commitment to recognising that impact looks, feels, and operates differently across disciplines and cannot be reduced to metrics at the ‘end’ of the journey. Joyfully we are seeing considerable energy in this direction – Responsible Metrics calling out bad practice in quantifying effects, drives towards impact literacy embraced by publishers, ‘knowledge mobilisation’ arguably overtaking ‘transfer’ and ‘exchange’ as more expressive terms of implementation. We still have a long way to go, but the more everyone ‘in the frame’ is equipped to judge how their work connects into impact, the fairer (and more effective) we will be.
Finding meaning with ‘users’, not just ‘transferring knowledge’
I personally dislike the word ‘users’, as it conjures up images of some transactional process whereby academics apply their research to a waiting horde and *TA-DA* the public cry tears of joy that their lives are better. I’m not sure quite what language is better (very happy to be advised!) but whatever we use, we need to shift away from simplistic views of impact-as-productisation and impact-as-consumption. Impact can and must always be measured at the level of the beneficiary/ies, in terms of what is meaningful for them. As we move forward, we’re seeing co-production and patient and public involvement weaved more substantially into research delivery, and this shift of focus away from blunt assessment currency can only be a good thing.
Disrupting simplistic, linear models of ‘ultimate’ impact
Logic models of impact are great (and endorsed fabulously elsewhere) but they can unintentionally reinforce an expectation of linearity. Input-output-outcome models routinely position impact as the ultimate endpoint of a research journey. Fine. Well not fine, actually, if that shifts assessment and thinking to so far down the line that it blinds us to those meaningful effects along the way. I’ve written before about the dangers of focusing only on ethereal and magical endpoints but if we can truly understand the iterative and co-produced nature of impact, we can make some very meaningful changes indeed.
Valuing and investing in impact professional development or ‘There are people that do this stuff? Like, willingly?’
I’m extremely proud to lead the ARMA Impact Special Interest Group, a group of over 750 research managers with a core, or connected focus on impact. This community of practice grows every year in both size and combined expertise, and offers the most incredible combined set of insights into what it’s like to ‘do impact’. Yet REF risks painting a simple one to one picture of academic (project) to impact, leaving brokers and research manager invisible in impact discourse. Impact needs a strong and valued community of practice, both in terms of dedicated impact professionals and the integration of impact into academic career development. If we don’t acknowledge the skills, efforts and expertise inherent to driving research into practice, quite how much impact can we really expect to have?
Impact does not, and never has equalled REF. Impact is not about assessment compliance, nor is it about demonstrating effects at scale at the expense of meaningful change of any size. We need to recognise impact may be driven by REF, elevated by REF, or conversely deprioritised if it doesn’t ‘fit the REF mould’.
Impact already exists beyond REF.
So perhaps the question shouldn’t be what impact looks like beyond REF, it’s what you’d do even if it wasn’t being measured.
*This blog accompanies a plenary at the Making a Difference: Impact in the Social Sciences’ Conference, April 19th 2018, St Anne’s College, Oxford*
6 thoughts on “Making a difference in impact….is there life beyond REF?”
Thanks for publishing this online Julie in an accessible way. Really good perspectives on the broader moral and scientific context of impact. As an independent academic with an interest in impact I can see that REF is generating interest and innovation across the globe, so in that sense it is doing a good job to raise awareness of the unacceptability of undertaking research without an eye to where it might be going and for what benefit. However the accountability aspect is rather negative, suggesting that all researchers (and research institutions) need to pull their socks up and do something more worthwhile. I prefer the idea of impact beyond REF as being like ‘dancing like no-one is watching’ i.e. you are only going to improve at impact if you try out new moves and reach out in sometimes unexpected ways. Thanks again. Elizabeth M Morrow
LikeLiked by 1 person
Hi Elizabeth. Many thanks for the comments. I completely agree re weighted expectations around accountability – to me a large part of the problem is the blunt implementation of an agenda which (to be fair) does try to account for disciplinary differences and doesn’t expect all research to have impact. Many other comments to make here but ultimately amount to ‘the impact agenda cannot and should not force inappropriate practice’. Nor should it be used as an unquestionable reason for demanding impact be sought.
I love your image of ‘new moves’ too – absolutely, how can we truly go anywhere ‘new’ if we’re not just exploring for exploring’s sake??! A difficult balance definitely, but one we have to strike.
Much more for us to do! But definitely feeling a huge investment in unpicking and finding new solutions across the sector. J
Great post Julie. The link to ‘Publishers embracing impact literacy’ is a dead link. Wanted to check out whether T&F got a mention and what you saw our competitors doing!
Hi David – how odd! Apologies – I’ve checked and repaired the link and seems to work but please let me know if not. This particular link goes to the Emerald Impact manifesto page – http://emeraldgrouppublishing.com/rwi/pdf/manifesto.pdf. T&F aren’t mentioned but I must confess I don’t know as much about their ‘impact literacy’. Very happy to hear more