Telescope bibliographies -- are statistics comparable?
Uta Grothkopf & Angelika Treumann, European Southern Observatory Library
esolib@eso.org
Introduction
Citation statistics become more and more important -- to evaluate scientists, measure the scientific output of institutions, determine the impact of journals, and so on. Many scientific institutions maintain bibliographies of articles published by their scientists and monitor references made to these publications; these figures are often used to evaluate the effectiveness of the institution.
In astronomy, one additional aspect beyond numbers of publications and citations is of interest -- the telescopes that were used to obtain the observational data on which articles are based. Observatories want to know how many articles staff and visiting astronomers have produced after observing runs on their telescopes. Increasingly, they are also being asked to compare their results to those of other institutions.
In the summer of 1999, we conducted a statistical study on the number of publications based on observations with ESO telescopes as well as with other major telescopes worldwide. During this study, it became obvious that compiling telescope bibliographies can mean different things to different institutions. If we want (or have) to compare statistics across observatories, we must pay utmost attention to how these bibliographies are compiled, or we will end up comparing apples to oranges. In the following, we describe possible inconsistencies in the way we identify our material.
1. Journal / publication coverage
Telescope bibliographies often are compiled by screening the journals and conference proceedings available in the observatory's library. Depending on the size, specialization, and available budget of the library, the range of accessible publications can vary considerably. Even in a discipline like astronomy with only a limited number of core journals, this can lead to differences in the final statistics.
2. Refereed / unrefereed publications
Institutions' bibliographies (general as well as telescope bibliographies) do not necessarily distinguish between refereed and unrefereed publications, but may list all papers published by the institute's scientists including publications in unrefereed journals, books, and conference proceedings. Although most telescope bibliographies include only refereed publications, we may encounter exceptions.
3. Selection policies
a) Identification of papers that are based, at least partly, on new data Publications based on observations can be subdivided into two groups: those presenting new, previously unreported data, and those that make use of already published data. Many papers refer back to published data, but also present new observations; these belong to the former group. In order to retrieve those papers that are based, at least partly, on new observational data, publications must be carefully examined.
Authors of papers based on new observational data should indicate clearly which telescopes were used to obtain them. While observations typically are acknowledged in the footnotes, specific telescopes and instruments often are mentioned in the "observations" section. In the course of the years we have also come across many examples where authors did not follow this pattern; telescopes can be mentioned almost anywhere in papers.
Usually, authors who used previously published data give a bibliographic reference. However, in some cases such bibliographic references given in conjunction with telescope observations refer just to an article describing technical features of an instrument. These papers must not be confused with those presenting observational data.
Some publications may require additional details in order to decide upon their inclusion in the bibliography. Resolving these cases by contacting authors for clarification or consulting underlying observing proposals is troublesome, and time constraints may prevent this thoroughness in many cases.
b) In- or exclusion of papers based on previously published data?
According to our survey conducted in mid-1999, the majority of astronomical institutions exclude from their telescope bibliographies those papers that don't present new data. However, this selection criterion is not universal and unfortunately there is no standard policy to which all observatories adhere. Some institutions don't even follow strict rules, but allow themselves some flexibility and make decisions on a case-by-case basis. This, of course, makes statistics comparison even more error-prone.
Like many other observatories, we at ESO are convinced that telescope bibliographies should include only those papers that present new data; otherwise publication lists would be swamped with unlimited numbers of follow-up papers analyzing already published observations. This policy categorically excludes all papers from our telescope bibliography that use previously reported data, for instance (published) catalog data or (published) data retrieved from the ESO data archive, as well as papers containing data analysis without presenting any new data.
c) Special cases: imaging surveys and archive data
In addition, institutions should set up policies on treatment of publications that make use of special data collections, for instance imaging surveys and archive data.
Astronomical long-term projects like imaging surveys may result in large sets of observational data. Papers originating from these surveys often are published in several parts, one or some of them presenting new observations, while other parts analyze the data. Following our policy, telescope bibliographies should only include those parts that present new data; most "chain papers" should be excluded.
Many observatories open their data archives to the scientific community, inviting astronomers to retrieve data on particular objects they are interested in. In order to distinguish between publications using archive data and those presenting new observations, it might be advisable to maintain them in a separate database. In any case, papers based on use of archive data should be clearly indicated in telescope bibliographies.
Conclusion
A number of possible inconsistencies were presented that can affect telescope bibliography compilation. As astronomical institutions become increasingly interested in comparing numbers of publications based on telescope observations, variations in compilation methods among observatories will cause confusion and may, finally, lead to wrong interpretations of statistics. We are therefore quite interested in initiating discussion on this topic and, if possible, agree on internationally comparable policies for telescope bibliography compilation. Your comments will be highly appreciated; please contact us at esolib@eso.org.
