Enhancing Web Access
to Library Catalog
The Niels Bohr Library once kept its catalogs of books, and the International Catalog of Sources for History of Physics and Allied Fields, on cards. These were converted to an electronic database in the 1980s and a Web interface was provided in 1997. The Library has recently achieved another milestone by getting its catalog open to searches though the major Internet search engines.
The Library's catalogs were previously part of the "deep Web." Potential users would have to know that the catalog existed, where to find it, and how to use it, before they could find any catalog record. Now, users can find catalog records in a familiar and simple interface, such as Google, that will lead them to the Library's textbooks, oral history interviews, archives, International Catalog of Sources records and other resources. (The Center's Web exhibits, the Physics History Finding Aids Web site, and the Emilio Segrè Visual Archives were already being indexed by Internet search engines.) Once users have thus found catalog records, they will be able to take advantage of the custom features of the Library's own Web-based search and browse tools.
This took nearly a year of effort to achieve. Center staff had to run a series of tests to understand how the search engines' Web crawlers gathered information about the catalog and how the catalog could be made to provide the desired information. Then, when we created a mechanism that unleashed these crawlers on the catalog, they consistently overwhelmed and crashed the catalog servers.
The Center was fortunately able to use an additional server that was no longer needed in the American Institute of Physics's server farm in Melville, NY. The single server, which had till then performed all the functions required to provide a Web catalog, was replaced with two one to run the database and another to run the Web interface to the database. Center staff then mirrored the entire contents of the database as Web pages. Search engines now crawl these mirror pages rather than the live catalog so that the catalog servers no longer crash under the load. A researcher landing on a mirror page is immediately forwarded to the corresponding record in the live catalog, providing access to all its features and functions. Finally, we slightly altered the Web page design of catalog records so that users finding them through Internet search engines, rather than the Center's own search interface, would know what they have found and where they have ended up.
As a result, use of the catalogs has increased by a factor of 5 to 10. In January of 2005, users viewed 114,000 catalog records through the Internet search engines. One consequence is increased use of the International Catalog of Sources leading researchers to resources in the history of physics held at archives other than AIP's. We have already had a researcher come to our own library, who would not have learned we hold a resource of value to her if it had not turned up in a Google search. This project is part of the Center's continuing efforts to find new ways of using the World Wide Web to preserve and make known the history of physics and allied sciences. Readers are invited to look for our catalog records in search engines such as Google. Did you know, for example, that Niels Bohr spoke on NBC radio (in English), and we have it on tape?