Information about http://www.ra-review.ac.uk/reports/roberts/roberts_summary.pdf
Review of research assessment Report by Sir Gareth Roberts to the UK…
Tags: assessing research, assessment practices, assessment report, chapter 3, evidence base, executive summary, gareth roberts, glossary of terms, group membership, invitation, policy environment, preface, radical overhaul, rae, resear, research assessment exercise, sir gareth, steering group, terms of reference, wholesale rejection,Pages: 21
Language: english
Created: Wed May 28 15:56:16 2003
Display cached document
Page 1
Page 2
Page 3
Page 4
Page 5
Page 6
Page 7
Page 8
Page 9
Page 10
Page 11
Page 12
Page 13
Page 14
Page 15
Page 16
Page 17
Page 18
Page 19
Page 20
Page 21
Review of research
assessment
Report by Sir Gareth Roberts to the
UK funding bodies
Issued for consultation May 2003
Contents
Preface: Sir Gareth Roberts 2
Executive summary 4
Figures 1-6 19
Chapter 1 Background to the review 21
Chapter 2 The review process 24
Chapter 3 The evidence base 26
Chapter 4 Our approach 30
Chapter 5 Proposed model 33
Chapter 6 Implementation 57
Annexes 62
A: Steering group membership and terms of reference 62
B: Policy environment 63
C: A guide to the 2001 RAE 67
D: Operational review of the 2001 RAE 71
E: Analysis of responses to the `Invitation to contribute' to 81
the review of research assessment
F: Assessing research the researchers' view 88
G: Changes in research assessment practices in other 91
countries
H: Additional costs implicit in the recommendations 95
I: Glossary of terms 100
1
Preface
I am immensely grateful to the funding councils for the opportunity to carry out a review of research
assessment. It has been both a great responsibility and a great pleasure.
The recommendations in this report constitute a radical overhaul of the Research Assessment
Exercise (RAE). They do not however represent a wholesale rejection of the RAE and the principles
upon which it was built. All who examine the impact of the RAE upon UK research and its
international reputation must, I think, agree that it has made us more focused, more self-critical and
more respected across the world. It has done this, in large part, by encouraging universities and
colleges to think more strategically about their research priorities.
I have not developed these proposals in a vacuum. They have been canvassed very widely across
the sector and its stakeholders. I am indebted to a great many people who have given up their time
to discuss my ideas and contribute to their development. Over the course of over 50 meetings, I
have been gratified by the positive response of almost all of those I have spoken to.
Nevertheless, I am conscious of one criticism which has been made a number of times. I have
proposed a system which appears more complex than what has gone before. It is a truism that what
is new appears more complex than what is familiar. However, I acknowledge a sense in which these
proposals do sacrifice simplicity for efficiency and fairness.
I believe it is time to move away from a `one-size-fits-all' assessment, to a model which concentrates
assessment effort where the stakes are highest. This would reduce the burden of assessment on
our universities and colleges but it does, inevitably, lead to a system which on paper appears more
complicated.
Throughout the review, I have been careful to respect the autonomy of each funding council and
indeed that of the territories they serve. It is my profound wish that the elements of the UK research
system will continue to grow together as a cohesive unit. However, I have made sure that these
proposals provide sufficient flexibility for each funding body to tailor its funding policy to meet the
needs of each nation.
The funding councils will also need to address the interface between the assessment process
described in this report and other sources of funding for research, in particular their own support for
work with business and the community and for the development of research in subjects without a
research tradition.
This report is being published by the funding councils as a consultation. I am very glad that this is
the case. The research community and its stakeholders have the opportunity not merely to read the
recommendations but to influence the funding councils in deciding whether and how to implement
them. I will watch progress with interest, in the knowledge that consultation can only improve the
proposals.
This report, then, is not the last word. Indeed, even were the consultation to produce no criticism,
there would still be work to do. I have only hinted in the report at some of the technical issues which
a new assessment exercise will have to resolve problems such as the division of the research
base into subject groupings for assessment (the `units of assessment'), the development of proxy
2
measures to help panels and institutions take decisions on the quality of research, or the
development of templates for my proposed assessment of research competences.
More important, I urge the funding councils to remember that all evaluation mechanisms distort the
processes they purport to evaluate. My team and I have tried to investigate the effects our proposals
will have upon the behaviour of managers in universities and colleges. Once the report is in the
public domain, it will become much easier to explore these behavioural consequences and I urge
the funding councils to do so thoroughly before taking any final decisions.
My acknowledgements, if complete, would exceed the report in length. I have already mentioned
those who have taken the trouble to share their thoughts on early versions of these proposals too
many to mention by name. They will, I hope, recognise the sincerity of my thanks.
I am indebted also to my excellent steering group, who have supported and challenged me in equal
measure. Both services are acknowledged with thanks.
Finally, I would like to acknowledge the invaluable help I have received from my team based at the
Higher Education Funding Council for England, particularly Tom Sastry, who acted as secretary to
the review and Vanessa Conte, as project manager. I salute their efforts.
Sir Gareth Roberts
Wolfson College, Oxford
3
Executive summary
The Research Assessment Exercise
1. The Research Assessment Exercise (RAE) provides ratings of the quality of research
conducted in universities and higher education colleges in the UK, to inform the selective allocation
of funds in accordance with the quality of the work undertaken.
2. The system was designed to maintain and develop the strength and international
competitiveness of the research base in UK institutions, and to promote high quality in institutions
conducting the best research and receiving the largest proportion of grant.
How the system works
3. The RAE is essentially a peer review process. In the last exercise in 2001, research in the UK
was divided into 68 subject areas or units of assessment. An assessment panel was appointed to
examine research in each of these areas.
4. Higher education institutions were invited to make submissions, in a standard format, to as
many units of assessment as they chose. There was no upper or lower limit on the number of units
an institution could submit to. Nor was there any limit on the number of staff submitted as research
active, although data were published on the proportion of staff submitted as research active.
5. In RAE2001 panels produced grades on a seven point scale (1, 2, 3a, 3b, 4, 5 and 5* five
1
star) . However, 80% of the researchers whose work was assessed were in submissions receiving
one of the three top grades (4, 5, and 5*), while 55% were included in submissions receiving one of
the top two grades (5 and 5*). The amount of discrimination provided by the exercise is therefore
less than the length of the rating scale would suggest.
Background to the review
6. Following the outcome of the 2001 RAE, the funding bodies decided that the RAE ought to be
reviewed in the light of the following concerns:
a. effect of the RAE upon the financial sustainability of research
b. an increased risk that as HEIs' understanding of the system becomes more
sophisticated, games-playing will undermine the exercise
c. administrative burden
d. the need to properly recognise collaborations and partnerships across institutions and
with organisations outside HE
e. the need to fully recognise all aspects of excellence in research (such as pure
intellectual quality, value added to professional practice, applicability, and impact within and
beyond the research community)
f. ability to recognise, or at least not discourage, enterprise activities
1
The same seven point grading scale was used in the previous exercise in 1996. Earlier exercises used shorter scales.
4
g. concern over the disciplinary basis of the RAE and its effects upon interdisciplinarity
and multidisciplinarity
h. lack of discrimination in the current rating system, especially at the top end with a
ceiling effect.
7. In June 2002 Sir Gareth Roberts, President of Wolfson College, Oxford was asked to review
research assessment on behalf of the UK higher education funding bodies. Sir Gareth's work has
been supported by a steering group and assisted by officers. This is the report to the funding bodies
by Sir Gareth and his team.
Key points
8. In the course of the review, we have reviewed some major strands of evidence:
a. 420 responses to our public `invitation to contribute'
b. an operational review of RAE2001 undertaken by Universitas higher education
consultants
c. report on international approaches to research assessment (update of 1999 study)
undertaken for the review by the Science Policy Research Unit at the University of Sussex
(`Report on responses')
d. a programme of nine workshops with practising researchers undertaken for the review
by RAND Europe
e. 44 informal consultative meetings with key stakeholders
f. open public meetings in Sheffield, Birmingham, Edinburgh, London, Cardiff and Belfast.
9. A number of themes have emerged strongly from each of these strands:
a. the importance of expert peer review
b. the need for a clear link between assessment outcomes and funding
c. the need for greater transparency, especially in panel selection
d. the need to consider carefully the trade-off between comparability of grades and the
flexibility for assessors to develop methods appropriate to their subject
e. the need for a continuous rating scale
f. the need for properly resourced administration of the RAE
g. consistency of practice across panels.
10. There are two main purposes of research assessment: to support the resource allocation
models of the funding councils, and to provide comprehensive and definitive information on the
quality of UK research in each subject area. We do not advocate pursuing one of the purposes of
the RAE to the exclusion of the other. However, we have in most cases come to regard the first
(informing funding) as more important than the second (providing quality information for a wide
variety of stakeholders).
11. This is a pragmatic view driven by the increasing costs of assessment and of research itself.
Assessing research to meet the limited requirements of the funding councils is a demanding enough
task for both the assessors and the assessed. Given the strains on the system, its costs, and the
importance of its decisions for the allocation of public funds, we lean towards the view that the
5
research assessment process should focus upon providing the information the funding councils
require to allocate those funds in a way which is fair, transparent and efficient.
12. We propose retaining many of the key features of the existing process:
a. a UK-wide system
b. dependence upon expert peer review to identify the best research
c. panel members recruited from within the research community (but not necessarily all
UK-based academics)
d. peer reviewers informed by performance indicators but not obliged to reflect them in
grading
e. an assessment organised on the basis of disciplinary panels
f. panels establish their own assessment criteria in consultation with their research
community
g. transparency: panel criteria and working methods are published years in advance of
the process
h. panels provide information on the quality and volume of research
i. a process designed to encourage institutions to make strategic choices about the areas
of research they prioritise
j. those who are assessed control their input into the process: submissions are put
together by institutions.
Our recommendations
13. We have taken RAE2001 as our starting point and made our recommendations in relation to
it. Our principal reforms could be summed up as follows:
a. the burden of assessment for institutions and assessment panels linked to the amount
of funds the institution is competing for
b. separate assessment of competences such as the development of young researchers
c. greater transparency, especially in panel selection
d. greater involvement of non-UK researchers
e. credible structures to ensure consistency of practice between panels.
f. flexibility for assessors to develop methods appropriate to their subject
g. grade bands abolished in favour of a profile of the research strength of each
submission, providing for a continuous rating scale
h. controls on the scores awarded, to prevent grade inflation
i. a clear link between assessment outcomes and funding
j. a properly resourced administration.
14. A summary chart showing the research assessment process which would be created if our
recommendations were accepted is included as figure 1.
Centrality of expert review
15. Some of us believed, at the outset of the process, that there might be some scope for
assessing research on the basis of performance indicators, thereby dispensing with the need for a
complex and labour-intensive assessment process. Whilst we recognise that metrics may be useful
6
in helping assessors to reach judgements on the value of research, we are now convinced that the
only system which will enjoy both the confidence and the consent of the academic community is one
based ultimately upon expert review. We are also convinced that only a system based ultimately
upon expert judgement is sufficiently resistant to unintended behavioural consequences to prevent
distorting the very nature of research activity.
Recommendation 1
Any system of research assessment designed to identify the best research must be based
upon the judgement of experts, who may, if they choose, employ performance indicators to
inform their judgement.
Frequency of the assessment
16. Research is an activity which requires a stable environment in which to flourish. The merits of
research often become apparent over many years and there is a strong ethic among researchers
which leads them both to strive for and to respect work of the highest quality. All of these factors
strongly suggest a credible (and relatively onerous) expert review assessment conducted at long
intervals.
17. With these considerations in mind, we have seriously considered a significant extension in the
assessment period from between three and five years, to eight or even ten years.
18. In the end, however, we have to be mindful of the right of government, as the ultimate funder
of research, to invest on the basis of up-to-date quality information. We recognise that there is a
need for reliable information on the performance of the research base if government is to compare
its claims for support with those of other public service areas.
19. Therefore we propose only a small increase in the assessment period, to six years. We also
propose that, at the mid-point of the cycle, the funding councils should monitor volume indicators.
The purpose of this monitoring would not be to re-assess the research, but rather to pick up
changes in the level of activity which might indicate that a department had been closed or its
research activity dramatically scaled back. Where this appeared to have occurred, the funding
council would have the option of investigating further. We would not recommend that the funding
councils make any revisions to grant levels unless there is evidence of a very significant fall-off in
research activity which could only be accounted for by significant disinvestment.
Recommendation 2
a. There should be a six-year cycle.
b. There should be a light-touch `mid-point monitoring'. This would be designed only to
highlight significant changes in the volume of activity in each unit.
c. The next assessment process should take place in 2007-8.
Assessment of research competences
20. Submissions to RAE2001 were expected to contain a statement of the research strategy and
environment (known as RA5). Panels were asked to produce a single grade reflecting not only the
7
quality of research output but also the features which underpin a unit's performance and its ability to
continue to perform. These features include its staffing policy, its treatment of young researchers,
and long-term financial planning as reflected in that statement.
21. We propose to separate the assessment of these `competences' from the assessment of
research quality in order to make it more visible and credible. The assessment would be based upon
sets of objective criteria related to specific actions. We suggest that these criteria would be grouped
under four headings (see figure 2):
a. research strategy (the coherence of an institution's research strategy including an
assessment of the credibility of its targets for obtaining funding)
b. development of researchers, including postgraduate research (PGR) students,
postdoctoral researchers and junior lecturers
c. equal opportunities policies and success in putting them into practice (this would relate
to an institution's policies for ensuring equality of opportunity for all its staff, not just those in
research roles)
d. dissemination of research beyond the academic peer group. This would cover an
institution's policy on encouraging a spectrum of activities, ranging from collaboration with
2
organisations outside HE, through the use of research to enhance teaching, and work
promoting the public understanding of research topics.
22. It would rest with the funding councils to decide what sanctions to take against an institution
failing the competence assessment. Should they wish to adopt a common approach, we would
propose that an institution failing its assessment against any one of the competences would be
allowed to enter the next research assessment, but would not receive funding on the basis of its
performance in that assessment until it had demonstrated a satisfactory performance. Given a two-
year period between the competences assessment and the main assessment, and a further year
between the assessment and the incorporation of its results into funding formulae, this would
provide the institution with a three-year period in which to improve before sanctions would be
enforced.
Recommendation 3
a. There should be an institution-level assessment of research competences, undertaken
approximately two years before the main assessment.
b. The competences to be assessed should be institutional research strategy, development
of researchers, equal opportunities, and dissemination beyond the peer group.
c. An institution failing its assessment against any one of the competences would be
allowed to enter the next research assessment but would not receive funding on the
basis of its performance in that assessment until it had demonstrated a satisfactory
performance.
2
We gave specific consideration to the use of research to inform teaching. It is self-evident that one of the ways in which
research organisations can attempt to ensure that their work has a positive impact upon the practice of others is by including
it in their own teaching or communicating it in a form which helps others to do the same. Therefore, if we are concerned with
encouraging a broader view of the dissemination of academic research, to exclude links with teaching would seem peculiar.
The QAA might be in a position to comment on the strength of those links.
8
Assessment burden in proportion to reward
23. At present all institutions and units are assessed in the same way. We believe that the full
weight of research assessment ought not to be brought to bear on all research, and that a lighter-
touch process may be appropriate for less research intensive institutions and units with less to gain
or to lose from the assessment process.
24. One crude measure of research intensity is the proportion of an institution's funding council
grant for teaching (T) and research (R) which is received for research: R/(T+R). It is beyond the
remit of the review to consider whether the funding councils should use this or any other metric as a
means of categorising institutions; and we have certainly not presumed that they will. We have used
it to explore the efficiency of research assessment as it affects institutions that are least dependent
upon funding council research grant to support their activities.
25. In 2002-3 there were 40 out of 132 English HEIs for whom R/(T+R) came to less than 2%.
These institutions received a total of £566 million in teaching funding and only £6.7 million in
research funding. They made 240 submissions to RAE2001, which yielded an average of £27,580 in
funding in 2002-3 compared to an average across the exercise of over £455,000 per submission.
For these institutions, therefore, and for the panels and administrators tasked with their assessment,
the RAE is over 16 times less efficient than the norm.
26. There will be those who argue that research assessment is not only about funding. We
recognise that the assessment of research is a valuable service which institutions use to benchmark
their progress. However, we believe it is increasingly difficult to provide this service where there is
no realistic prospect of funding.
27. We therefore propose a three-track assessment process:
a. option of a separate approach for the least research intensive institutions
b. assessment by proxy measures against a threshold standard (Research Capacity
Assessment or RCA) for the less competitive departments in the remainder of institutions
c. expert review assessment similar to the old RAE for the most competitive departments
(Research Quality Assessment or RQA).
28. A key feature of our proposals is that institutions would be asked to take decisions on the
work they wish to submit for the full RQA at the level of the subject area rather than the individual.
This means that, in submitting to RQA they would forfeit the right to submit staff from that area to the
RCA and any funding consequent upon RCA results.
Recommendation 4
a. There should, in principle, be a multi-track assessment enabling the intensiveness of the
assessment activity (and potentially the degree of risk) to be proportionate to the likely
benefit.
b. The least research intensive institutions should be considered separately from the
remainder of the HE sector.
9
c. The form of the assessment of the least research intensive institutions would be a matter
for the relevant funding council.
d. The less competitive work in the remainder of institutions should be assessed by proxy
measures against a threshold standard.
e. The most competitive work should be assessed using an expert review assessment
similar to the old Research Assessment Exercise.
Outputs of the assessment
29. In RAE2001 and its predecessors, panels produced grades on a seven point scale (1 to 5*).
The system was, we believe, a qualified success. The grades commanded respect and helped to
drive an improvement in UK research which has helped to maintain the UK's position as one of the
leading research nations.
30. However, in recent years, doubts have begun to be expressed about the grading system.
Concerns include the following:
a. the consequences of gaining or losing a grade are so great that institutions are obliged
to `play games' in order to ensure that they fall the right side of the grade boundary
b. it is extremely difficult to ensure that grades are comparable across subject areas,
especially subject areas between which there is little overlap
c. there is a risk that the grade improvement seen in 2001 may create an expectation of
improvement from one exercise to the next, which might itself drive grade inflation
d. grades attempt to capture the absolute performance of departments, although funding
must of necessity be dependent upon relative performance. The outcry provoked by the
funding councils' inability to `fund the results' of the 2001 RAE shows how this can create a
gap between expectation and reality.
31. We propose an alternative system which we believe offers a solution to these problems. For
each submission the panel would produce a `quality profile' (see figure 3). For each submission the
panel would decide how much work could be defined as meriting one, two or three stars (or no
stars). This system emphasises that the focus of the assessment is to discriminate between the very
good research and the very best. It also provides for what is, in effect, a continuous grading scale,
eliminating the problems created by grade boundaries.
32. We do not propose that panels formally attach a score to named individuals. It should be a
point of principle that we should not report on individuals' performance on the basis of an
assessment which they cannot choose whether or not to enter and which considers a sample of
their work which they do not themselves select. If a panel chooses to use `researcher-level' analysis
to inform its judgement, we suggest that such analyses should not in any circumstances be retained
or disclosed. If it is not possible to avoid retaining and disclosing such analyses, we suggest that
they should not form a part of any panel's working methods. Research assessment should remain
an assessment of institutional research quality within a subject area, rather than a review of the
performance of individuals.
10
33. To enable institutions to be advised in advance of the funding weights to be employed by
each funding council, to prevent grade inflation and to guarantee the integrity of ratings, we propose
that panels should be given guidelines on the proportions of three star, two star and one star ratings
which should be awarded in the absence of evidence that the subject outperformed other subjects
when measured against international benchmarks. For example, the funding councils might decide
that x% of the work submitted should be rated as three star work. These guidelines should be the
same for each unit of assessment. A moderation process would be built into the panel structure to
establish whether a panel should be able to depart from the guidelines.
34. This would enable the funding councils to provide something for which there is an
overwhelming demand: clear information on the relationship between the scores achieved in the
assessment and the impact upon funding (see figure 4). It is clearly desirable, though not absolutely
necessary, if the weights employed by each funding council are consistent. Funding weights will
undoubtedly vary between subjects, there being a rationale for greater selectivity in subjects where
there is already a large amount of research of the highest class and where the costs of remaining
competitive are particularly high.
Recommendation 5
a. The output of the Research Quality Assessment should be a `quality profile' indicating
the quantum of `one star', `two star' and `three star' research in each submission. It will
not be the role of the assessment to reduce this profile to summary metrics or grades.
b. As a matter of principle, star ratings would not be given to named individuals, nor would
the profile be published if the submission was sufficiently small that individual
performance could be inferred from it.
c. Panels would be given guidelines on expected proportions of one star, two star and
three star ratings. These proportions should be the same for each unit of assessment. If
they awarded grades which were more or less generous than anticipated in the
guidelines, these grades would have to be confirmed through moderation.
d. The funding councils should provide institutions with details of the relative value, in
funding terms, of one star, two star, and three star research, and of research fundable
through the Research Capacity Assessment in advance of the assessment. These ratios
might vary between disciplines.
A panel structure designed to ensure consistency
35. Even if panels are limited in the number of `star ratings' they can award, there is still a need
for a mechanism to ensure consistency of practice in order to:
a. ensure that panel criteria are allowed to diverge where there is a rationale for them to
do so, and are consistent where this has a bearing upon the integrity of the assessment
b. ensure that panels adhere to their own criteria
c. enable the process as a whole to provide feedback to the funding councils on areas of
research which are stronger or weaker than the norm in relation to international benchmarks.
11
36. We therefore propose a hierarchical panel structure, in which the recommendations of each
sub-panel would be signed off by a higher level body with a remit to ensure consistency of practice.
Our proposed structure is illustrated in figures 5 and 6.
Recommendation 6
a. There should be between 20 and 25 units of assessment panels supported by around 60
sub-panels. Panels and sub-panels should be supported by colleges of assessors with
experience of working in designated multidisciplinary `thematic' areas.
b. Each panel should have a chair and a moderator who would sit on each sub-panel. The
role of the moderator would be to ensure consistency of practice across the sub-panels
within the unit of assessment.
c. Each panel should include a number of non-UK based researchers with experience of
the UK research system.
d. The moderators of adjacent panels should meet in five or six `super-panels' whose role
would be to ensure consistency of practice between panels. These `super-panels' should
be chaired by senior moderators who would be individuals with extensive experience in
research.
Respecting disciplinary differences
37. Whilst we are convinced that there is a need to ensure that panels' practices are consistent
(or equivalent) where this is appropriate, we consider it equally important to define those aspects of
the assessment where greater sensitivity to disciplinary differences would enhance the reliability of
the results. We have therefore brought forward proposals which we believe will enable panels to
assess work in their fields in a way which is both more sensitive and more consistent.
Recommendation 7
a. The rule that each researcher may only submit up to four items of research output
should be abolished. Research Quality Assessment panels should have the freedom to
define their own limits on the number and/or size of research outputs associated with
each researcher or group.
b. Research Quality Assessment panels should ensure that their criteria statements enable
them to guarantee that practice-based and applicable research are assessed according
to criteria which reflect the characteristics of excellence in those types of research in
those disciplines.
38. We propose to work alongside the research communities to develop a set of discipline-
specific performance indicators which could form the basis of the indicative bandings. These
bandings would be produced at least one year before the assessment. They would inform
institutions' key strategic choice which subject areas to submit for RQA and which for RCA. They
would also inform the decisions of the panels but would not bind the panels in any way.
12
39. The number of bands would be allowed to vary between subjects, reflecting the extent to
which different subject communities were prepared to accept performance indicators as a reliable
indicator of quality.
Recommendation 8
a. The funding councils should work alongside the subject communities and the research
councils to develop discipline-specific performance indicators.
b. Performance against these indicators should be calculated a year prior to the exercise
and institutions advised of their performance relative to other institutions.
c. The weight placed upon these indicators, as well as their nature, should be allowed to
vary between panels.
Submission rules
40. It is important to define what is assessable as well as how it is assessed. As noted above,
institutions submitting to RQA would forfeit the right to submit staff from that area to the RCA and
any funding consequent upon RCA results. The RAE defines the population of eligible researchers
in a unit of assessment, and uses this information to publish the proportion of staff submitted. Under
our proposals it will be important to ensure that this information is reliable, and to minimise the
scope for artificially defining less research active staff as belonging to a unit of assessment with
which they have little to do. This may well require stronger audit procedures.
41. The abolition of grades should lower the stakes for institutions. Top-rated researchers would
attract funding even if the average score was depressed by the inclusion of others in the return. We
are confident that this would reduce `games-playing'.
42. Nevertheless it remains important that RQA results present an accurate picture of the
strength of a department whilst at the same time providing the flexibility needed to protect teaching
staff from being pressurised into prioritising research. To this end, we propose that at least 80% of
staff in any sub-unit of assessment must be included in any RQA return.
43. The consequences of this are much less significant than they would have been had an 80%
minimum been introduced in RAE2001. In that exercise, the presence of less well regarded
researchers in the submission could depress the grade awarded. It would therefore affect the
recognition and potentially the funding received by the strong researchers present in the
submission. With the introduction of the quality profile this will not happen. The amount of high
quality research in the submission will be clearly visible, and we anticipate that the funding councils
will wish to reward this irrespective of the amount of less excellent work contained in the same
submission.
44. We also consider it to be extremely important to encourage institutions to make collaborative
submissions. We advise the funding councils to do everything possible to facilitate this.
13
45. It should be possible for research groups to be submitted as a single entity. This would enable
qualified staff whose research activity has contributed to important work for which they have not
been formally recognised to be included in the assessment.
Recommendation 9
a. Where an institution submits to Research Quality Assessment in a sub-unit of
assessment, all staff in that sub-unit should become ineligible for the Research Capacity
Assessment, even if they are not included in the Research Quality Assessment
submission.
b. The funding councils should establish and promote a facility for work to be submitted as
the output of a group rather than an individual where appropriate.
c. The funding councils should consider what measures could be taken to make joint
submissions more straightforward for institutions.
d. Where an institution submits a sub-unit of assessment for Research Quality Assessment
no fewer than 80% of the qualified staff contracted to undertake research within the sub-
unit of assessment must be included in the submission.
e. All staff eligible to apply for grants from the research councils should be eligible for
submission to Research Quality Assessment.
Research strategy
46. In RAE2001 panels had the opportunity to consider statements on the research strategy and
environment underpinning each submission. Under our proposals, the elements contained within
that statement would be covered in the research competences assessment.
47. Notwithstanding the institutional competences assessment, we anticipate that panels would
benefit from receiving a research strategy statement. This would indicate the institution's plans for
research at unit level. Panels would be able to see the institutional research strategy and ought to
be able to confirm whether the two documents were consistent with one another.
48. Panels would be asked to report on the strategies, indicating any they considered inadequate
or exemplary. It would be left to the funding councils to decide whether to act on these reports.
Recommendation 10
Each panel should consider a research strategy statement outlining the institution's plans for
research at unit level.
Supporting emerging units
49. We have proposed that institutions be allowed to nominate a minimal number of emerging
units. They would have to demonstrate their commitment to developing research in these
departments to the level where it was genuinely competitive with top-rated research. The relevant
14
assessment panels would propose objective success criteria which would indicate that they were
catching up with leading departments. Progress against these criteria would be assessed at the mid-
point of the assessment period, by officers of the research assessment process.
Recommendation 11
There should be a facility for institutions to identify emerging units and a mechanism for
evaluating their progress after three years.
Links to other funding processes
50. We suggest to the funding councils that the results of the RQA could be used to identify
suitable candidates to compete for monies made available for the following purposes:
a. `partnerships of excellence' which would recognise the sharing of excellence with other
top-rated research units. These are seen as a means of helping the funding councils to
encourage collaboration.
b. third stream activities, to improve the interactions between HE and business, the public
3
sector and the wider community .
Recommendation 12
The funding councils should consider the extent to which data produced by the research
assessment process can be used to inform other funding processes, including third stream
funding and partnerships of excellence.
Panel recruitment
51. The recruitment of panel members was considered by institutions to be one of the less
transparent aspects of the exercise. In part, we believe this was because institutions did not engage
with recruitment as closely as with other parts of the process which were more salient to them. The
most transparent way to identify assessors would be to advertise each post. However, over 1,500
people contributed to the assessment of research in RAE2001, and the burden of open competition
for that number of people would, we believe, be excessive.
52. We therefore propose a package of measures intended to balance transparency with the
need to ensure that the burden is in proportion to the benefit obtained.
Recommendation 13
a. Job descriptions and person specifications should be produced for Research Quality
Assessment panel and sub-panel members, chairs and moderators as well as senior
moderators and the chair of the exercise. These should be published before steps are
taken to fill the posts.
b. Nominations of panel members should be sought from stakeholders in the same way as
in RAE2001.
3
The wider community can be taken to include the voluntary sector and, of course, society as a whole.
15
c. The chairs' and moderators' posts for the main Research Quality Assessment panels
should be advertised, and candidates should be chosen by a selection panel as should
the senior moderators.
d. Sub-panel chairs should be elected by the membership of the outgoing panels, from a
shortlist not necessarily confined to previous RAE panel members.
e. Panel members and sub-panel members should be chosen by sub-panel chairs, panel
chairs and moderators on the basis of their fit with the published job description and
person specification.
f. The funding councils should monitor and report upon the gender balance of sub-panel
members, sub-panel chairs, panel chairs, moderators and senior moderators.
Administrative reforms
4
53. The operational review of the RAE has found that the RAE team itself performed admirably.
However, it could justly be said that the planning of the administrative support for RAE2001 failed to
anticipate the demands it would face.
54. The review's conclusion is stark. Resources for RAE2001 were inadequate. Even if there
were no changes to the exercise, the central administration would require increased resources.
`The pressures imposed by the timetable for the assessment phase, the workload on
key players (panel members and secretaries, the RAE team) and the demand for
several of the supporting services, ran a high risk of major disruption, though none
occurred. The same degree of dedication and commitment which all those involved
showed cannot be assumed for any similar further exercise. More staff (or funds to
outsource services) would be required; and all inputs should be realistically costed
and paid for.'
55. It follows that to provide an adequate and safe service to the same specification as RAE2001
would require more resources in real terms than the £5-6 million devoted to RAE2001.
56. Furthermore, we believe that there is scope to relieve the burden on the rest of the sector by
providing a more comprehensive service from the centre.
Recommendation 14
a. The research assessment administration should employ full-time panel secretaries who
would each work with several panels.
b. The senior moderators (see recommendation 6) should not be external to the RAE
administration in the same way as RAE panel chairs. They should be accountable for the
successful administration of the exercise as well as for its results, should be employed
4
`An operational review of the 2001 RAE' (Universitas 2003).
16
by the funding councils, and should be in post at an early stage in the process.
c. The funding councils ought to consider the burden imposed upon their staff and
resources by the need to support the RAE and ensure this is properly accounted for. In
some cases this may involve embedding functions within the RAE administration itself.
d. The funding councils should recognise that the cost of specialist advice is likely to be
greater in a future exercise than it was in RAE2001.
Further work
57. When the funding councils present their own proposals for research assessment for
consultation, these will need to be accompanied by a full assessment of their impact upon equality
of opportunity for all groups of staff, and the burden placed by the assessment upon institutions,
assessors and their own administrative capacity. The funding councils will also need to assure
themselves that proposals do not require panel members of the funding councils and their
employees to accept any unnecessary legal risks. There will, in addition, be a need to investigate
the behavioural consequences of our proposed reforms.
58. The strands of work mentioned above are necessary in order to enable the funding councils
to take fully informed decisions on the adoption of our recommendations once a full public
consultation has taken place. In addition, there are strands of work relating to the implementation of
the recommendations, which, while less relevant to that decision point, need to be progressed
urgently if, as some have suggested, the next assessment is to take place in 2007. In particular, the
funding councils may wish to move quickly to:
· identify units of assessment
· develop discipline-specific metrics
· develop templates for our proposed assessment of research competences.
Recommendation 15
The funding councils should undertake or commission further work in parallel with the consultation
on these proposals to ensure that proposals for research assessment taken as a whole:
a. do not create, encourage or facilitate discrimination on the grounds of age, sexual orientation,
5
political belief, disability, gender, race or religion
b. do not create any unnecessary legal risks for the funding councils or the panel members
c. do not create excessive or unnecessary burdens upon panel members, institutions or the
funding councils
d. are not likely to have behavioural effects which the funding councils consider unacceptable.
5
Additionally, the funding councils may wish to seek advice on the best ways of assessing
institutions' policy and practice on equal opportunities, drawing upon the experience of other
sectors.
17
Interdependence of our proposals
59. Many of our proposals have been designed to complement one another. There is a risk that
an emphasis upon the impact of individual recommendations rather than the generality of the
proposals may lead to pressure to take forward a package which is incoherent. It is important
therefore that the implications of revisiting any one of our proposals upon the feasibility of others,
are properly understood.
Recommendation 16
Should the funding councils ultimately decide to pursue some but not all of our
recommendations, members of the review team and the steering group should be
reconvened to advise on the feasibility of the revised package of reforms.
18
Figure 1 Sexennial research assessment
SEXENNIAL RESEARCH ASSESSMENT
Assessment of institutional competencies
Institutional Competencies
Collation of discipline-based metrics or other specific indicators
Collation of discipline based metrics other specific indicator
Institutional INSTITUTIONAL Units of
assessment
Assessment SUBMISSION Assessment
Research Research
Quality Capacity
Consultation with Expert Strategy and metrics
Strategy & Metrics
funding Bodies
Fundingbodies review
Review based analysis
Block Grant
grant
funding
Funding
Figure 2: Institutional competences
Item Exemplar of good Satisfactory Unsatisfactory
practice
Research strategy
Development of
researchers
Equal
opportunities
Dissemination of
research
Figure 3: Volume determination for funding
Sub-panel A B C D E Totals
Grade
***
**
*
Totals
19
Figure 4: Funding ratios (factors shown here are illustrative)
Unit of Research Quality funding factor Research Capacity
Assessment funding factor
unstarred
*** ** *
X 6 4 0 0 1
Y 6 4 2 0 1
Z 4 3 2 0 1
Figure 5: Model structure
RAE2001 Proposed RQA structure
Chair
Umbrella panels (5)
Super-panels (5)
RAE panels (60) RQA panels (20-25)
Sub-panels (50-70)
Sub-panels (26)
Colleges of
advisers in
designated areas
Specialist
advisers (464)
Other specialist
advice
Figure 6: Research assessment panels
Super- Engineering RAE Chair
Arts &
panels & Physical Life Sciences Etc
Humanities
Sciences
Panels Physical Senior
Engineering Mathematics Etc Moderator
Sciences
Sub- Moderator
panels Mechanical Electrical Chemical Etc
20