Bibliometric data is not without its limitations. It is extremely difficult to assess scholarly performance, and in particular the quality of the research, through numbers and figures alone. For this reason, it is essential to take multiple indicators into consideration when conducting bibliometric analysis. Ideally, these indicators will be determined in accordance with the respective subject discipline and in agreement with the scientist in question.

Example: Number of Publications

The number of publications produced (research output) provides the research community with an overview of the productivity of an individual or institution. However, the number of publications does not automatically guarantee the quality of research output or scholarly performance. Regular publication activity does not always mean that good quality work is consistently being produced. The quality of a publication is reflected in its ‘impact’, which includes its informative value and its significance within its respective field. This cannot be conveyed through figures or through Journal Impact Factor rankings.

The value of bibliometric data depends heavily upon the quality and focus of the database used. The following example is a case in point:

Stephen Hawking achieved an H-Index of 59 in the Web of Science database, compared with an index of 76 in Google Scholar; however, in Scopus, he obtained an index of 19.

At present, there is no robust bibliometric database which covers all research areas or which can be used consistently across subject areas. For this reason, researchers are advised to agree on a suitable database together before bibliometric analysis is conducted.

Bibliometric indicators are often applied to make interdisciplinary comparisons.  However, it is not advisable to make direct comparisons across subject areas as scholarly publication cultures can differ greatly from field to field. Instead, the aim should be to customize indicators for use within each specific field.

Over the last few years, a practice known as ‘strategic publication behavior’ has increased, and ‘bibliometric optimization’ has also been observed. Journal citation cartels (often referred to as the practice of ‘citation stacking’), ‘salami slicing’ (the splitting of research findings into multiple publications) and other forms of bibliometric manipulation are no longer isolated cases.

New regulations introduced by the German Research Foundation (Deutsche Forschungsgemeinschaft, or ‘DFG’ for short) illustrate the extent to which bibliometric manipulation has already progressed. The new DFG regulations have been in place since 2010 and are applied to funding proposals and final reports. The DFG’s aim is to reduce the flood of publications in research and the pressure on researchers to publish. The DFG President Matthias Kleiner, addressed these issues in a statement in February 2010:

“Whether in performance-based funding allocations, postdoctoral qualifications, appointments, or reviewing funding proposals, increasing importance has been given to numerical indicators such as the H-index and the impact factor. The focus has not been on what research someone has done but rather how many papers have been published and where. This puts extreme pressure upon researchers to publish as much as possible and sometimes leads to cases of scientific misconduct in which incorrect statements are provided concerning the status of a publication. This is not in the interest of science.”