The trouble with research impact

An interesting piece from Mark Henderson, on the Times Online website.  Lord Drayson, Science Minister, was defending the case for research impact to young researchers opposed to its introduction in the REF. .” Lord Drayson argued that if science and academia are to continue to claim significant public funding, they have to do a better job of explaining its benefits — not least to make the case to an often sceptical Treasury on the look-out for easy cuts.”  Meanwhile, at the Royal Society, Henderson tells us that Lord Rees was making an eloquent case against the impact agenda, saying that: “There’s a risk that current efforts to prioritise and ‘audit’ academic research will backfire, by eroding the strength of our universities and thereby weakening the UK’s competitiveness as a high-tech nation.” Read the article online and watch the debate with Lord Drayson online at the THES.

Research Impact Assessment – Examples from RAND Europe

Since RAND Europe have now produced their report for HEFCE on international practices in assessing the impact of research, it may be a good time to also point out some of the work that RAND has been doing in this area and to look at examples of their work in assessing research impact for a range of various international clients. Those who are interested can follow this link to the RAND website.

Where to publish? Who to cite? The behaviour of researchers. The influence of assessment.

The Research Information Network (RIN) recently published a report on the effects of measures to evaluate the worth of  research upon the publishing and citing behaviour of academic researchers. The report, “Communicating knowledge: how and why UK researchers publish and disseminate their findings“, finds that many are confused as to what the various funding bodies expect of them in terms of communicating their research.  Most would appreciate better guidance as to how this will impact upon any assessment of their work.  

  • What factors influence decisions on the timing of publication and dissemination of research? Is it better to publish in the high status journal or to communicate more directly with the people most interested in the topic of the research? 
  • How do patterns vary across the disciplines?
  • What place have the perceived requirements for research assessment occupied in the full range of factors that have influenced publication and citation behaviour? 

A variety of methods, including focus groups and online questionnaires, were used to consider these and other questions. The report found that many reserachers are apparently considering citing their colleagues’ work more often because of the introduction of bibliometrics. Some researchers also said they felt they were being pressurised into publishing too much, too soon.

HEPI response to REF proposals

On the 15th October, HEPI (Higher Education Policy Institute) published their response to the HEFCE proposals for the Research Excellence Framework. On the whole, HEPI is supportive of the proposals but is critical of plans to assess the impact of research. The introduction of impact assessment – i.e. non-academic impact – is something new and experimental. HEFCE are proposing 25% of the assessment score should be given to “impact”. The HEPI response views this as rather high given the experimental nature of the “impact assessment” element.
The report is available to download.

Brief guide to REF proposals

A document outlining the proposed format of the REF has been published at http://www.hefce.ac.uk/ref. The brief guide in PDF format appears near the bottom of the screen.
It summarises details such as how many outputs each person is likely to have to submit to the REF, the method of assessment, which subject areas are likely to use citation analysis, etc.

You are the REF

ResearchResearch and Evidence have developed a tool which can help you to explore what the new metrics system likely to be used for the REF might mean for your department. HEFCE has said that the metrics it is likely to be using are research income, citations and PhDs. You are the REF lets you explore different weightings for different factors to see how you come out.
Please note that HEFCE has not signed up to this tool – it is purely for exploration purposes.
You can find the tool at http://youaretheref.researchresearch.com/

HEFCE Director addresses Australians on research assessment

David Sweeney, Director of Research & Innovation at HEFCE, was quoted in The Australian newspaper yesterday as saying that research funds in the UK are being stretched too much to cover the volume of research in academia. “Our strategy is to improve the quality of research, not the quantity,” he is reported to have said.

Research Impact Assessment – a measure whose time has come?

For those interested in the idea of evaluating the impact of research and the issues this raises, a recent article by Claire Donovan (ANU & Chair of the Technical Working Group on Research Impact for the RQF) published in 2008  in New Directions for Evaluation, is well worth a read. In this article, “The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research“, she discusses the development of  the RQF’s  impact rating scale. The Australians at that time planned to go beyond quantitative mechanisms measuring “investment from industry, commercialization, and  technology transfer”  to a broader definition of impact which included social, economic, environmental and cultural benefits. Donovan discusses the conflicting influences of government policy, keen to make academic research more business and industry focussed, with the concerns of academics to protect pure research.  The period in which impact is measured was one issue of concern. The benefit of pure research is not always quickly realised.

In the end, the change of government in Australia led to the RQF & the research impact element being dropped. However, David Sweeney of HEFCE was quoted in yesterday’s AUSTRALIAN: “There are some bits we’ve pinched,” he says. “You were doing impact explicitly. You chose not to (continue). I understand why because you have other ways of doing that. But in our environment we thought it was worth trying.”

TIMES take on HEFCE REF proposals

An article in the TIMES claims that HEFCE’s new proposals (now out to consultation) will put an end to “days of university researchers developing formulas for the perfect cheese sandwich” and other such “irrelevent research”. This is how the article interprets the effect of the research impact assessment element of the REF. David Sweeney of HEFCE is quoted as saying that: “The Research Excellence Framework will recognise and reward excellent research and sharing new knowledge to the benefit of the economy and society, and will ensure effective allocation of public funds”. Critics see this as a threat to academic freedom. See the full article at Timesonline