HEFCE publishes RAND Europe report on Research Impact Methodologies

News that HEFCE have just published Capturing Research Impacts: A review of international practice is of particular interest for Swansea University since we will be participating in the HEFCE Pilot Exercise on Research Impact.

This is the report Hefce commissioned from RAND Europe to bring together knowledge gained from international experiences in assessing research impact. The review will help inform development of this new aspect of the REF.  Notably, the review “suggests that the work of the Australian RQF Working Group on Impact Assessment might provide a basis for developing an approach to impact in the REF”. (See previous entries on this blog regarding the RQF.)

RAND Europe has already established experience in this area. See for example, Exploring the impact of arthritis research

Full report and executive summary are available at: http://www.hefce.ac.uk/pubs/rdreports/2009/rd23_09/

Research Impact – debate, policy, methods

There has been much debate about this element of the REF proposals. Nobel laureates have raised objections. The Russell Group have stated that they are “broadly supportive of introducing a measure of the economic and social impact of research, provided that this is underpinned by a robust methodology which commands the confidence of the academic community”.  A petition against the use of research impact in the allocation of funding has been created on the No. 10 website.  An earlier petition, set up in June 2009, before the REF proposals were published, opposed the imposition of impact statements which funding councils now often require as part of a grant application; and knowledge transfer staff in universities have complained that they are being swamped with requests to write these “impact statements”.

Sir Alan Langlands, chief executive of the Higher Education Funding Council for England, and David Sweeney, Director of Research & Innovation at HEefce, have both indicated that research funding is limited and that there is a need to evaluate the impact of research. The policy context they cite is the Science & Innovation Investment Framework, 2004-14. There is a need, say Hefce, to “strengthen links between investment in research and the economic and social benefits it delivers”.  The same message is to be found in the Research Councils Economic Impact Group’s 2006 paper, Increasing the Economic Impact of the Research Councils.

HEFCE has set out  proposed methods for evaluating impact and is running a pilot exercise.  The methodology to be employed in evaluating research impact,  seems  similar to that proposed for the now defunct Australian RQF exercise. The Australian top 8 research intensive universities opposed impact evaluation, which they felt would be overly bureaucratic and time-consuming. On the other hand,  universities represented by the Australian Tehnology Network, were supportive of the proposals.  The ATN produced an interesting brief report based upon their experience of the Australian research impact pilot exercise. It discusses methodology, and case study examples.

Here in the  UK, UNICO, the Knowledge Transfer professionals’ organisation, have proposed the development of a metrics system to measure the impact of UK publicly funded research. These metrics, they suggest, could be used in attracting industry funding for research and to encourage international research collaborations with universities and with industry.

HEFCE Director addresses Australians on research assessment

David Sweeney, Director of Research & Innovation at HEFCE, was quoted in The Australian newspaper yesterday as saying that research funds in the UK are being stretched too much to cover the volume of research in academia. “Our strategy is to improve the quality of research, not the quantity,” he is reported to have said.

Research Impact Assessment – a measure whose time has come?

For those interested in the idea of evaluating the impact of research and the issues this raises, a recent article by Claire Donovan (ANU & Chair of the Technical Working Group on Research Impact for the RQF) published in 2008  in New Directions for Evaluation, is well worth a read. In this article, “The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research“, she discusses the development of  the RQF’s  impact rating scale. The Australians at that time planned to go beyond quantitative mechanisms measuring “investment from industry, commercialization, and  technology transfer”  to a broader definition of impact which included social, economic, environmental and cultural benefits. Donovan discusses the conflicting influences of government policy, keen to make academic research more business and industry focussed, with the concerns of academics to protect pure research.  The period in which impact is measured was one issue of concern. The benefit of pure research is not always quickly realised.

In the end, the change of government in Australia led to the RQF & the research impact element being dropped. However, David Sweeney of HEFCE was quoted in yesterday’s AUSTRALIAN: “There are some bits we’ve pinched,” he says. “You were doing impact explicitly. You chose not to (continue). I understand why because you have other ways of doing that. But in our environment we thought it was worth trying.”

TIMES take on HEFCE REF proposals

An article in the TIMES claims that HEFCE’s new proposals (now out to consultation) will put an end to “days of university researchers developing formulas for the perfect cheese sandwich” and other such “irrelevent research”. This is how the article interprets the effect of the research impact assessment element of the REF. David Sweeney of HEFCE is quoted as saying that: “The Research Excellence Framework will recognise and reward excellent research and sharing new knowledge to the benefit of the economy and society, and will ensure effective allocation of public funds”. Critics see this as a threat to academic freedom. See the full article at Timesonline

Lammy on REF & economic impact of research

The Higher Education Minister, in a speech to Universities UK Conference last week, said that he wanted the REF to send “a strong signal and give a strong financial incentive” for university departments to “find ways of helping turn that [research] into impacts that benefit the economy and society as a whole”. In terms of how impact may be assessed, he said: “Improved products and services for business are important, but so are things like improved quality of life, and better evidence-based Government policy-making”. Read the THE report of this speech or read the Minister’s speech in full.

REF – Assessing the Impact of Research

“Research impact” is going to play a significant role in the forthcoming REF. So how will they measure the social, economic, and cultural impacts of university research? HEFCE have invited English HEIs to join a pilot exercise to help develop proposals for assessing the impact of research.  The pilot should run from autumn 2009 to mid-2010. Submissions for assessment will consist of three main types: case studies “illustrating the research driven contribution to economic, social, public policy, cultural or quality of life impacts”; a statement “summarising the full range of impacts for the submission as a whole”; and “supporting quantitative indicators”.

REF & Bibliometrics

HEFCE have recently published: “Interim report of the REF bibliometrics pilot exercise”.  The full report is due out Autumn 2009, but this makes interesting reading. It looks at possible models, and some of the limitations of using bibliometrics. It would seem that the REF Expert Advisory Groups see bibliometrics more as a “useful aid to expert review”, and that this might vary according to the discipline.