A very interesting new article has just been published by researchers in Italy, who have analysed data from the first Italian national research evaluation to see whether there is a correlation between bibliometric indicators and peer review.
The questions they posed were:
- Are peer review judgements and (article and journal) bibliometric indicators independent variables?
- What is the strength of the association?
- Is the association between peer judgement and article citation rating significantly stronger than the association between peer judgement and journal citation rating?
They found “a compelling body of evidence that judgements given by domain experts and bibliometric indicators are significantly positively correlated” and in their conclusions, they suggest:
Bibliometrics are not independent of peer review assessment. The correlation between peer assessment and bibliometric indicators is significant but not perfect. Peer review should be integrated with bibliometric indicators in national assessment exercises.
Franceschet, M & Constantini, A (2010) “The first Italian research assessment exercise: A bibliometric perspective“, Journal of Informetrics, in press, corrected proof Science Direct [Online].
HEFCE have published “Analysis of data from the pilot exercise to develop bibliometric indicators for the REF – The effect of using normalised citation scores for particular staff characteristics”
The report analyses the data from the pilot exercise to develop bibliometric indicators for the REF. It analyses the effect of using bibliometrics (citation scores) in the REF upon certain types of research staff. For example, early career researchers will be less likely to have many citations. It also looks at age and sex, gender, ethnicity, disability of researchers as well as those who are part-time staff.
The report recommends:
If citation data are used then the four UK higher education funding bodies will need to ensure that institutions planning to make submissions to the REF are aware of the results of this analysis so that they can take them into account when selecting staff for inclusion. Further, panels will also need to account for the differences found and will require guidance as part of their equality briefing.
This session for academic staff and researchers will look at the ways you can monitor the impact your publications are having, especially when considering the REF. We will also look at Researcher ID as a way of raising your own profile, Journal Citation Reports, and Web of Science facilities such as cited reference searching, citation mapping, times cited, citation alerts and amending your author name if necessary.
The session will be held in the Library & Information Centre (Room – PC3) on 15th February 2011 from 12.30pm to 1.30pm.
To book a place, please contact firstname.lastname@example.org
A list of the Main and Sub-Panel chairs is now available on the HEFCE website. The process of appointing panel members will now begin and these are expected to be announced in the new year.
The full reports from the Impact Pilot Study are available online from HEFCE. As well as the report, there are example case studies from Clinical Medicine, Earth Systems & Environmental Sciences, Physics, Social Work & Social Policy and English Language & Literature.
In a brief press release on their website, RCUK “welcomes the findings of the Research Excellence Framework (REF) pilot scheme that demonstrates it is possible to assess the economic and societal benefits arising from research and how this can be achieved across all disciplines.
Professor Dave Delpy, RCUK Impact Champion said: “RCUK is committed to supporting excellent research that ensures social wellbeing and economic prosperity and ultimately places the UK in a position of leadership on the world stage of research and innovation. We support the wide definition of impact, as set out in this report, which includes social, economic, cultural, environmental, health and quality of life benefits. This reflects the approach already adopted by RCUK through Pathways to Impact.”
RCUK has worked closely with the Funding Councils to ensure that proposals for the REF will be effective in pursuing shared objectives and will continue to support the REF by working with HEFCE to contribute expertise in developing impact assessment methodologies.”
It was announced today that panel members for the REF are now being recruited. Applications are needed by 17 September. Details of how to apply are available at: http://www.hefce.ac.uk/research/ref/pubs/2010/01_10/.
The site also contains more detail about the subject coverage of the REF subpanels and the document “Units of assessment and recruitment of expert panels” gives an indication of the work involved in being a panel member.