Role of Peer Review in Assessing the Value of Basic Research Reaffirmed

“Dominant reliance on quantitative measures...will at best distort assessments and more likely will prove destructive as research proposals and funding decisions are optimized for the measures rather than for the best and most exciting science.” --Richard Zare, Stanford University

These days, federal agencies are under increased pressure from Congress and the Administration to ensure that federal programs provide value to the American public for every dollar paid in taxes. The Government Performance and Results Act of 1993 (GPRA) is one attempt to encourage accountability. It calls for federal agencies to develop, by the end of fiscal year 1997, multi-year strategic plans and metrics for assessing progress toward agency goals.

Chairman Robert Walker (R-PA) of the House Science Committee on July 10 invited representatives from government and industry to discuss the challenge of evaluating the programs of civilian agencies that perform or fund basic science research. This is research for which, according to the committee’s hearing “the primary goal is to produce new knowledge, with long-term and, frequently, unpredictable outcomes.”

While the witnesses were unanimous in agreeing upon the value of GPRA to help prioritize and manage federal research efforts, they concurred that performance in basic research cannot be evaluated strictly by quantitative measurements. Richard Zare of Stanford University (and incoming chairman of the National Science Board) reported that recent efforts by the National Research Council to quantitatively measure various aspects of scientific research “produced a negative result. That is, we were unable to use reliably and with any confidence quantitative measures to assess the health of a field.... Our work very much affirmed this Committee’s observation that `quantitative measures may not be feasible for basic research.’”

Alternatively, the witnesses stressed that the best method of assessing the quality and value of long-term fundamental research was through the judgment of independent panels of experts - in essence, merit-based peer review. Representatives from the National Science Foundation, the Department of Energy, NASA, NIST, NOAA and EPA described how they used peer review panels or external advisory committees to prioritize and evaluate research programs. All emphasized the importance of GPRA language allowing flexibility if, as stated in the hearing charter, “it is not feasible to express performance goals in an objective, quantifiably measurable form.”

Ernest Moniz, the Office of Science and Technology Policy’s Associate Director for Science, described efforts by the Administration “aimed at laying a common foundation for assessing fundamental science programs.” Because of the difficulty in determining the ultimate benefit to the nation from a research investment, Moniz suggested that the government use an intermediate, or enabling, goal for the purposes of measurement: “leadership across the frontiers of scientific knowledge.” This goal ensures that the U.S. is positioned, he said, “to capitalize on scientific advances anywhere” in the world, in order to benefit from those advances. Moniz, too, reiterated that “merit review based on peer evaluation must remain the primary vehicle for assessing the excellence and conduct of science at the cutting edge.”