Administration to Develop Criteria to Evaluate Applied and Basic Research

“To reform government, we must rethink government” is the lead sentence of an August 24 Office of Management and Budget report entitled “The President’s Management Agenda” (http://www.whitehouse.gov/omb/budget/fy2002/mgmt.pdf ). A component of this agenda is the development of an “objective investment criteria for federal R&D projects” that will evaluate several DOE applied R&D programs. Separate criteria will be developed for evaluating basic research programs.

These R&D criterion are one of the nine program initiatives that the White House says “address the most apparent deficiencies where the opportunity to improve performance is the greatest.” The section entitled “Better R&D Investment Criteria - Department of Energy” is about two and one-half pages long. It begins by stating that science and technology are “critically important,” and “as a result, every federal research and development (R&D) dollar must be invested as effectively as possible.”

OMB faults the conduct of federal R&D on several grounds. It states that “the ultimate goals of this research need to be clear.” “Vague goals lead to perpetual programs achieving poor results,” OMB charges, criticizing NASA’s space science program goal of charting our destiny in the solar system and the USGS goal of providing science for a changing world. The report cites the lack of evaluation criteria to measure the effectiveness of R&D investment, and to determine what these investments have produced. Measurements are rarely linked to funding decisions. “Without this information, decisions about programs tend to be made on the basis of anecdotes, last year’s funding level, and the political clout of local interest groups,” the report states.

Another criticism is that “Many R&D projects have ended up stepping beyond the legitimate purposes of government to complete with - or unnecessarily subsidize - commercial ventures.” In addition, “many R&D projects directly benefit corporations that could fund their own R&D projects without federal assistance.”

To correct this problem, OMB and DOE are developing performance criteria for several applied R&D programs. These criteria will be used to determine the FY 2003 budget for DOE programs in the following areas: Solar and Renewable Energy, Nuclear Energy, Clean Coal, Fossil Energy, and Energy Conservation. Input is being sought from other White House offices, federal R&D agencies, research management experts, “and groups with an interest in the federal R&D portfolio to improve investment criteria and their implementation.”

OMB expects that the efficiency of these programs will increase by at least 10 percent through the application of these evaluation criteria. This efficiency “will be measured by expected dollar per barrel of oil avoided/produced and dollar per ton of pollution reduced,” with criteria “tailored to the specific objective of the research.” Programs that benefit individual firms, instead of entire sectors, will be reduced by not less than 50 percent. Additional program performance criteria are described.

Regarding other applied and basic research, OMB writes, “After our initial effort in applying uniform investment criteria to the applied energy technology programs, OMB will assist in the transfer of investment criteria to the rest of DOE, and other Departments and applicable agencies with applied R&D programs in time to assist in the formulation of the President’s 2004 Budget. OMB and the Office of Science and Technology Policy will also work with NASA, the National Science Foundation, the Department of Defense, the National Institutes of Health, and DOE to develop separate criteria, to be issued in Spring 2002, for evaluating basic research during formulation of the 2004 Budget.”

Evaluation of federal programs was made more systematic by the Government Performance and Results Act, passed during the Clinton Administration. It was widely recognized that evaluating basic research would be difficult. The efforts by the Bush Administration to tie an evaluation methodology directly to budget formulation for FY 2004 warrant attention.