GAO Finds No “Old-Boys’ Network” in Peer Review System

Over the years the peer review system for awarding federal research funds has come under fire for being an “old-boys’ network,” made up of prominent academics from elite institutions channeling funding back to their institutions and colleagues, and biased against lesser-known schools and regions. Members of Congress often use this criticism to justify earmarking funds for their constituencies. Chairman of the Senate Governmental Affairs Committee, John Glenn (D-Ohio) asked the General Accounting Office (GAO) to study the issue of equity in the peer review process. While GAO did find some weaknesses in the system at the three agencies it examined (NSF, NIH and the National Endowment for the Humanities), it concluded that those criticisms most commonly made by foes lack credibility.

The 133-page report, dated June 24, 1994 and entitled, “Peer Review: Reforms Needed to Ensure Fairness in Federal Agency Grant Selection,” looked at fairness in three areas: 1.) the selection of reviewers and how well they represent the pool of grant applicants; 2.) the scoring of proposals by reviewers; and 3.) the final funding decisions of the agencies. This FYI will focus on findings relevant to NSF.

GAO sampled 50 successful and 50 unsuccessful proposals from five programs within NSF: biochemistry; economics; math, algebra and number theory; electrical engineering; and theoretical physics. The report concludes that “Overall, peer review processes appear to be working reasonably well.” It notes that “several factors alleged to affect reviews were not, in fact, related to scores at any of the agencies, [including] the applicant’s region, academic rank, or even employing department’s prestige.”

Contrary to allegations about an elitist system, GAO found “little evidence to support the notion that peer review panels are staffed disproportionately from among researchers at a handful of elite institutions. On the contrary, among our sample cases those from elite institutions tended to be somewhat underrepresented among reviewers compared to applicants.” NSF program officers reported that researchers from elite institutions were more likely to decline requests to serve as reviewers.

The study found that while women are overrepresented on peer review panels at NSF, they are underrepresented on external reviews. Among regions, only the South Atlantic was found to be underrepresented among reviewers, and contrary to allegations, New England was not among the three highest of the nine regions for reviewers. Professional age of reviewers (years since PhD) generally followed the distribution among applicants, except for the most recent graduates, which the report found were “greatly underrepresented” as reviewers.

GAO cites several weaknesses in the proposal rating system: “Agencies do little to ensure that reviewers have an accurate and similar understanding of the agency’s criteria and rating scales,” and “reviewers often relied on unwritten or informal criteria when evaluating proposals,” such as inclusion of preliminary results.

Reviewers whose own work was least proximate to the proposals were likely to give better ratings. This could variously be interpreted to mean that researchers with more relevant expertise were better able to detect flaws, or were downgrading competitors. On average, the more prestigious the applicant’s department was perceived to be, the better the rating received. Proposals from applicants known directly to reviewers got the best scores on average, while unknown applicants “fared worst.”

GAO also finds an apparent relation between race and score: on average, whites received better scores than non-whites. While NSF does not provide reviewers with information on the applicant’s race or ethnicity, the report admits that some information might be discernable based on knowledge of the applicant or institution. It cannot be determined, the report states, whether the relationship is the result of bias or the quality of proposals. NSF responds, in comments on the report, that it “has long-standing policies, programs and outreach efforts that are expressly designed to encourage women, minorities and persons from other than elite universities to apply and, other things being equal, to decide in their favor.” The Foundation also reports that if all NSF-funded research programs are considered, “NSF’s award rates for women and minority applicants are roughly comparable to those for men.”

While the report finds certain apparent trends, it cautions that the data is insufficient to determine whether they represent actual bias or not. “Much of the variation in peer review scores of proposals,” GAO concludes, “cannot be explained by the factors we could measure.... We have interpreted this to mean that scores are driven by agencies’ criteria, including the importance of the issue and the quality of the research design.”

Although NSF gives its program officers considerable discretion in choosing which proposals to fund, the report found proposal scores “the dominant independent variable” in funding decisions. Proposals with weaker scores were more likely to be funded if less money was requested, and the larger the request, the less likely the proposal was to be funded if the researcher was not well-known. As GAO points out, “this result can be interpreted as showing prudence on the part of NSF program officers.”

Proposals perceived as being from top schools were twice as likely to be funded as others; while applicants best known to reviewers were three times as likely to be funded as unknowns.

The report offers the following general recommendations to improve the peer review process: address underrepresentation of young/junior scholars as reviewers; increase monitoring of discrimination, and test by comparing blind and conventional reviews; develop a rating system where criteria are rated separately as well as overall; and identify or formalize commonly-used unwritten rules.

In addition, it provides several NSF-specific suggestions: increase the use of panels over external reviews; more closely monitor inclusion of women and minorities among external reviewers; and increase efforts to calibrate ratings among reviewers.

One copy of the report, GAO/PEMD-94-1, can be obtained free of charge from GAO, P.O. Box 6015, Gaithersburg, MD 20884-6015, or by phone (202-512-6000) or fax (301-258-4066.)