Role of Peer Review in Assessing the Value of Basic Research

FYI
The American Institute of Physics Bulletin of Science Policy News
Number 106: July 12, 1996

Reaffirmed

"Dominant reliance on quantitative measures...will at best distort
assessments and more likely will prove destructive as research
proposals and funding decisions are optimized for the measures
rather than for the best and most exciting science."
     --Richard Zare, Stanford University

These days, federal agencies are under increased pressure from
Congress and the Administration to ensure that federal programs
provide value to the American public for every dollar paid in
taxes.  The Government Performance and Results Act of 1993 (GPRA)
is one attempt to encourage accountability.  It calls for federal
agencies to develop, by the end of fiscal year 1997, multi-year
strategic plans and metrics for assessing progress toward agency
goals.  

Chairman Robert Walker (R-PA) of the House Science Committee on
July 10 invited representatives from government and industry to
discuss the challenge of evaluating the programs of civilian
agencies that perform or fund basic science research.  This is
research for which, according to the committee's hearing "the
primary goal is to produce new knowledge, with long-term and,
frequently, unpredictable outcomes."

While the witnesses were unanimous in agreeing upon the value of
GPRA to help prioritize and manage federal research efforts, they
concurred that performance in basic research cannot be evaluated
strictly by quantitative measurements.  Richard Zare of Stanford
University (and incoming chairman of the National Science Board)
reported that recent efforts by the National Research Council to
quantitatively measure various aspects of scientific research
"produced a negative result.  That is, we were unable to use
reliably and with any confidence quantitative measures to assess
the health of a field.... Our work very much affirmed this
Committee's observation that `quantitative measures may not be
feasible for basic research.'"

Alternatively, the witnesses stressed that the best method of
assessing the quality and value of long-term fundamental research
was through the judgment of independent panels of experts - in
essence, merit-based peer review.  Representatives from the
National Science Foundation, the Department of Energy, NASA, NIST,
NOAA and EPA described how they used peer review panels or external
advisory committees to prioritize and evaluate research programs. 
All emphasized the importance of GPRA language allowing flexibility
if, as stated in the hearing charter, "it is not feasible to
express performance goals in an objective, quantifiably measurable
form."

Ernest Moniz, the Office of Science and Technology Policy's
Associate Director for Science, described efforts by the
Administration "aimed at laying a common foundation for assessing
fundamental science programs."  Because of the difficulty in
determining the ultimate benefit to the nation from a research
investment, Moniz suggested that the government use an
intermediate, or enabling, goal for the purposes of measurement:
"leadership across the frontiers of scientific knowledge."  This
goal ensures that the U.S. is positioned, he said, "to capitalize
on scientific advances anywhere" in the world, in order to benefit
from those advances.  Moniz, too, reiterated that "merit review
based on peer evaluation must remain the primary vehicle for
assessing the excellence and conduct of science at the cutting
edge."

###############
Audrey T. Leath
Public Information Division
American Institute of Physics
fyi@aip.org
(301) 209-3094
##END##########                                            

[an error occurred while processing this directive]