- Email: [email protected]
Evaluating science—performance assessment in research
The GPRA In 1993, the US Congress passed the Government Performance and Results Act (GPRA) to improve coordination among research agencies, Congress, and the White House and develop a more effective allocation process for research funds. It was designed to add credibility to the distribution of research dollars by removing historical and political factors and making it entirely performance based. GPRA attempts to do at the research-agency level what grant study sections do at the individualinvestigator level. Agencies receiving federal research funds are required to develop a five-year strategic plan summarizing their aims. The plan must include specific annual goals, a discussion of how these goals relate
Evaluating science – performance assessment in research
MEETING REPORT
Economic realities have put constraints on budgets everywhere. In science and technology sectors, these limits, coupled with ever greater numbers of researchers, have led to increased competition, and demands for accountability and relevance in research. Scientists and the government are obliged to find ways to evaluate and justify research expenditures. The American Association for the Advancement of Science (AAAS) participates actively in a range of science-policy areas. Their 23rd Annual Science and Technology Policy Colloquium* discussed the details of the 1999 research budget, annual projections, the changing roles of scientists in government, and an important piece of legislation that has come into effect recently.
Amy Muhlberg to the long-term plan and how the aims will be implemented. Congress can then score agency proposals and rank them for funding priority. The most controversial part of GPRA, and of research assessment in general, is the requirement for methods to measure performance in research. There is concern that the larger goals of science will be lost to quantitative measures of progress. It is not obvious how to evaluate or forecast progress in basic research on a yearby-year basis. GPRA does make some provisions, but no excuses, for the unpredictability of research. As Joshua Gotbaum of the Office of Management and the Budget summarized, ‘Research is different, but it is not exempt.’ Assessment elsewhere The unique nature of research is a problem faced by funding-allocation systems worldwide. The UK has been using performance assessment to prioritize funding for many years. The process has gone through many changes and is still maturing. Ben Martin of the Science Policy Research Unit, University of Sussex, UK, discussed the present system and
its history to show that not only could an assessment programme be developed but it could succeed. He concluded by suggesting further improvements, most notably calling for a greater involvement of the research community in the evaluation process. Performance assessment is here to stay, and the scientific community has an important role to play in ensuring that it is successful in allocating funds appropriately (see Box 1). One issue that remains unsettled is risk assessment. Failure is an integral part of the scientific process; can it be incorporated productively into strategic planning? ‘Playing it safe’ could result in research that progresses but lacks strong leadership and innovation. The assessment process needs to include a method of measuring risk, while still underlining its importance in the scientific endeavour. The discussions at the meeting highlighted that performance evaluation in research is still developing and that, as more factors are added to the equation, the system will hopefully evolve into one that meets the needs of science, funding agencies and the taxpaying public.
*AAAS Science and Technology Policy Colloquium – R&D: Getting Our Money’s Worth; Washington, DC, USA; 29 April – 1 May 1998.
BOX 1 – FURTHER INFORMATION For more information about science policy, visit: Asia: Links to pages maintained by various Asian governments. http://www.uow.edu.au/crp/links.htm#Asia Australia: Australian Academy of Science http://www.science.org.au/policy/policy.htm l Centre for Policy Research on Science and Technology (Simon Fraser Univ.): A terrific site, including contact information for faculty members. http://mot.cprost.sfu.ca/cprost/
http://www.sta.go.jp/welcome-en.html
Europe: WEST (European Web on Science and Technology) hosts meetings and electronic forums about European science and technology issues. http://www.west.it/ Japan: Science and Technology Agency. This is an English-language site sponsored by the Japanese government.
trends in CELL BIOLOGY (Vol. 8) August 1998
Copyright © 1998 Elsevier Science Ltd. All rights reserved. 0962-8924/98/$19.00 PII: S0962-8924(98)01322-1
Amy Muhlberg is at The Scripps Research Institute, La Jolla, CA 92037, USA. E-mail: amym@scripps. edu
335
Evaluating science – performance assessment in research
MEETING REPORT
Economic realities have put constraints on budgets everywhere. In science and technology sectors, these limits, coupled with ever greater numbers of researchers, have led to increased competition, and demands for accountability and relevance in research. Scientists and the government are obliged to find ways to evaluate and justify research expenditures. The American Association for the Advancement of Science (AAAS) participates actively in a range of science-policy areas. Their 23rd Annual Science and Technology Policy Colloquium* discussed the details of the 1999 research budget, annual projections, the changing roles of scientists in government, and an important piece of legislation that has come into effect recently.
Amy Muhlberg to the long-term plan and how the aims will be implemented. Congress can then score agency proposals and rank them for funding priority. The most controversial part of GPRA, and of research assessment in general, is the requirement for methods to measure performance in research. There is concern that the larger goals of science will be lost to quantitative measures of progress. It is not obvious how to evaluate or forecast progress in basic research on a yearby-year basis. GPRA does make some provisions, but no excuses, for the unpredictability of research. As Joshua Gotbaum of the Office of Management and the Budget summarized, ‘Research is different, but it is not exempt.’ Assessment elsewhere The unique nature of research is a problem faced by funding-allocation systems worldwide. The UK has been using performance assessment to prioritize funding for many years. The process has gone through many changes and is still maturing. Ben Martin of the Science Policy Research Unit, University of Sussex, UK, discussed the present system and
its history to show that not only could an assessment programme be developed but it could succeed. He concluded by suggesting further improvements, most notably calling for a greater involvement of the research community in the evaluation process. Performance assessment is here to stay, and the scientific community has an important role to play in ensuring that it is successful in allocating funds appropriately (see Box 1). One issue that remains unsettled is risk assessment. Failure is an integral part of the scientific process; can it be incorporated productively into strategic planning? ‘Playing it safe’ could result in research that progresses but lacks strong leadership and innovation. The assessment process needs to include a method of measuring risk, while still underlining its importance in the scientific endeavour. The discussions at the meeting highlighted that performance evaluation in research is still developing and that, as more factors are added to the equation, the system will hopefully evolve into one that meets the needs of science, funding agencies and the taxpaying public.
*AAAS Science and Technology Policy Colloquium – R&D: Getting Our Money’s Worth; Washington, DC, USA; 29 April – 1 May 1998.
BOX 1 – FURTHER INFORMATION For more information about science policy, visit: Asia: Links to pages maintained by various Asian governments. http://www.uow.edu.au/crp/links.htm#Asia Australia: Australian Academy of Science http://www.science.org.au/policy/policy.htm l Centre for Policy Research on Science and Technology (Simon Fraser Univ.): A terrific site, including contact information for faculty members. http://mot.cprost.sfu.ca/cprost/
http://www.sta.go.jp/welcome-en.html
Europe: WEST (European Web on Science and Technology) hosts meetings and electronic forums about European science and technology issues. http://www.west.it/ Japan: Science and Technology Agency. This is an English-language site sponsored by the Japanese government.
trends in CELL BIOLOGY (Vol. 8) August 1998
Copyright © 1998 Elsevier Science Ltd. All rights reserved. 0962-8924/98/$19.00 PII: S0962-8924(98)01322-1
Amy Muhlberg is at The Scripps Research Institute, La Jolla, CA 92037, USA. E-mail: amym@scripps. edu
335