NSTC Report Addresses the Challenge of “Assessing Fundamental Science”
As federal budgets grow ever tighter, the themes of accountability and efficiency with the taxpayers’ money are receiving increased attention. By fiscal year 1999, all federal departments and agencies will be required to produce a report assessing progress toward their stated goals, as called for in the Government Performance and Results Act (GPRA) of 1993 (see FYI #106.)
While some federal programs are amenable to judgment by quantitative measures, it is widely recognized that the performance of the basic research enterprise cannot be evaluated so easily. Outcomes and impacts of basic science are frequently not apparent for decades, and can come in unexpected ways and unpredictable areas. With the help of the Office of Science and Technology Policy (OSTP) and the National Science and Technology Council (NSTC), the government agencies that perform science have implemented pilot programs to develop metrics and methods for judging the government’s performance of fundamental research. The data accumulated so far are described in a July 1996 report by the NSTC’s Committee on Fundamental Science, entitled “Assessing Fundamental Science.”
According to the report, a central issue is defining a goal against which progress can be measured. Because basic research contributes to myriad national goals in a complex way and over long time periods, the Administration has identified an intermediate, or enabling, goal against which to assess progress: U.S. leadership across the frontiers of scientific knowledge. If the U.S. can remain at the forefront in all areas of science, it will be well-positioned to achieve other national goals such as improved health, environmental preservation, economic prosperity, national security, and quality of life. This objective, the report says, will provide “the principle yardstick for GPRA assessment strategies for fundamental science programs.”
The report admits that assessment techniques for fundamental science are still in their infancy, but concludes from experiences so far that “merit review based on peer evaluation will continue to be the primary vehicle” for assessing excellence. It cautions that pre-existing quantitative metrics are not sufficient to measure such aspects of basic science as innovation, and warns against indicators that might discourage risk-taking and creativity. The report sets out the following principles for evaluation: clearly-defined program goals; criteria that will encourage excellence and responsiveness; performance indicators that are useful and appropriate; avoidance of burdensome and counterproductive assessments; incorporation of merit review and peer evaluation; multiple sources and types of evidence; experimentation to develop effective assessment tools; reports that will inform and refine policy development; and results that can be understood by policymakers and the public.
“The passage of GPRA offers scientists and science managers the opportunity to adapt the best planning and management methods to build world-class science programs,” the report concludes. The Appendices discuss progress in pilot programs underway in NSF, DOE, NIST, NIH and USDA. The report, which runs 71 pages including appendices, can be found on Internet at: www.nsf.gov/sbe/srs/ostp/assess/start.htm
