American Institute of Physics
SEARCH AIP
home contact us sitemap

FYI Number 65: June 6, 2002

Update on Evaluation of R&D Programs

Last fall the Administration announced its intention to develop evaluation investment criteria for applied and basic research. The Office of Management and Budget and the Department of Energy made initial efforts to apply an evaluation methodology to Solar and Renewable Energy, Nuclear Energy, Clean Coal, Fossil Energy, and Energy Conservation programs (see 2001 FYI #117). While the implementation of these evaluation criteria for the FY 2003 budget request was more difficult than first envisioned, the Administration and Congress are moving ahead on the application of evaluation criteria to federal R&D programs.

The most recent evidence of the Administration's determination is a May 29 announcement in the Federal Register of the formation of a Performance Measurement Advisory Council. OMB will establish a six-member panel of experts in performance measurement "who are outstanding in their professional field and who are objective." "Council members will draw upon their expertise in creating, implementing and evaluating performance measurement standards and will make recommendations regarding the types of measures and benchmarking systems that departments and agencies can employ most effectively to track program performance." The council will exist for nine months.

Judging by a hearing of the House Appropriations Subcommittee on Interior and Related Agencies earlier this spring, appropriators, DOE officials, the OMB, and the National Research Council appear to be on the same page about the feasibility of program evaluation. Ranking Minority Member Norman Dicks (D-WA) explained that the subcommittee had contacted the National Academy of Sciences in 1999 to determine if the $22 billion of taxpayers' money invested in energy research in the last two decades was "worth it." The resulting report identified significant economic benefits from the research. Dicks explained that the subcommittee had worked since then with NAS, OMB, and DOE to develop a benefit measurement methodology that would be broader in scope.

Robert Fri, chairman of the committee that conducted the study, testified about the application of a methodology that was developed and applied to 39 case studies of energy efficiency and fossil energy programs. Fri testified that the committee found that "it is feasible to design and to apply consistently a uniform methodology to a diversity of applied research programs." Fri explained that future evaluations should be made of the research system, and not the progress of the research programs.

Next to testify was Mark Everson, Comptroller of OMB. He told the subcommittee that "Understanding the actual return on our federal investment dollar is all the more essential in the current environment, when we need to ensure that national security needs are fully met. We cannot afford to increase funding for all programs." He added, "Measuring the performance of R&D activities is one of the toughest challenges we face in bringing accountability to our investment decisions and linking funding to program performance. . . . We cannot, however, simply abandon efforts to assess the value of our $100 billion a year R&D investment." Everson stated that the Administration was working with experts and stakeholders to refine the R&D criteria that will be used in the FY 2004 budget process. Everson informed the subcommittee that he had spoken with OSTP Director John Marburger, who said that the pilot test had given the Administration experience with the evaluation process, and that it can move forward.

A joint statement by David Garman, DOE Assistant Secretary for Energy Efficiency and Renewable Energy and Carl Michael Smith, Assistant Secretary for Fossil Energy, was supportive of evaluation. "This Administration is strongly committed to improving DOE's research and development performance management system," they told the subcommittee.

A later statement by Garman and Smith summarized the current status of the evaluation of R&D programs: "The President and Secretary Abraham have made it very clear: improving our methodologies for estimating benefits and using these tools to develop better performance-based program strategies is one of the Department's highest priorities. It will also be one of the most complex and challenging analytical efforts the Department has ever undertaken."

Richard M. Jones
Media and Government Relations Division
American Institute of Physics
fyi@aip.org
(301) 209-3095

Back to FYI Home