Welcome!
To use the personalized features of this site, please log in or register.
If you have forgotten your username or password, we can help.
My Menu
Saved Items

Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation

Michel Zitt1   Contact Information, Suzy Ramanana-Rahary2 and Elise Bassecoulard3

(1) Lereco, Inra, Lereco, INRA, Nantes, France
(2) Observatoire des Sciences et des Techniques (OST), Paris (France)
(3) Lereco, INRA, Nantes, France

Summary  As citation practices strongly depend on fields, field normalisation is recognised as necessary for fair comparison of figures in bibliometrics and evaluation studies. However fields may be defined at various levels, from small research areas to broad academic disciplines, and thus normalisation values are expected to vary. The aim of this project was to test the stability of citation ratings of articles as the level of observation - hence the basis of normalisation - changes. A conventional classification of science based on ISI subject categories and their aggregates at various scales was used, namely at five levels: all science, large academic discipline, sub-discipline, speciality and journal. Among various normalisation methods, we selected a simple ranking method (quantiles), based on the citation score of the article in each particular aggregate (journal, speciality, etc.) it belonged to at each level. The study was conducted on articles in the full SCI range, for publication year 1998 with a four-year citation window. Stability is measured in three ways: overall comparison of article rankings; individual trajectory of articles; survival of the top-cited class across levels. Overall rank correlations on the observed empirical structure are benchmarked against two fictitious sets that keep the same embedded structure of articles but reassign citation scores either in a totally ordered or in a totally random distribution. These sets act respectively as a 'worst case' and 'best case' for the stability of citation ratings. The results show that: (a) the average citation rankings of articles substantially change with the level of observation (b) observation at the journal level is very particular, and the results differ greatly in all test circumstances from all the other levels of observation (c) the lack of cross-scale stability is confirmed when looking at the distribution of individual trajectories of articles across the levels; (d) when considering the top-cited fractions, a standard measure of excellence, it is found that the contents of the 'top-cited' set is completely dependent on the level of observation. The instability of impact measures should not be interpreted in terms of lack of robustness but rather as the co-existence of various perspectives each having their own form of legitimacy. A follow-up study will focus on the micro levels of observation and will be based on a structure built around bibliometric groupings rather than conventional groupings based on ISI subject categories.

Fulltext Preview (Small, Large)
Image of the first page of the fulltext


Export this article
Export this article as RIS | Text
 
Referenced by
12 newer articles

  1. Billaut, Jean-Charles (2009) Should you believe in the Shanghai ranking? : An MCDM view. Scientometrics
    [CrossRef]
  2. Agarwal, Pankaj (2009) Can literature analysis identify innovation drivers in drug discovery?. Nature Reviews Drug Discovery 8(11)
    [CrossRef]
  3. Ramanana-Rahary, Suzy (2009) Aggregation properties of relative impact and other classical indicators: Convexity issues and the Yule-Simpson paradox. Scientometrics 79(2)
    [CrossRef]
  4. Ball, Rafael (2009) Creation of journal-based publication profiles of scientific institutions — A methodology for the interdisciplinary comparison of scientific research based on the J-factor. Scientometrics
    [CrossRef]
  5. Levitt, Jonathan M. (2009) The most highly cited Library and Information Science articles: Interdisciplinarity, first authors and citation patterns. Scientometrics 78(1)
    [CrossRef]
  6. Radicchi, F. (2008) Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences 105(45)
    [CrossRef]
  7. Leydesdorff, Loet (2008) A global map of science based on the ISI subject categories. Journal of the American Society for Information Science and Technology
    [CrossRef]
  8. Glänzel, Wolfgang (2008) Subfield-specific normalized relative indicators and a new generation of relational charts: Methodological foundations illustrated on the assessment of institutional research performance. Scientometrics
    [CrossRef]
  9. Zitt, Michel (2008) Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology
    [CrossRef]
  10. Adams, Jonathan (2008) Calibrating the zoom — a test of Zitt’s hypothesis. Scientometrics
    [CrossRef]
First | Next | Last
Remote Address: 38.107.191.112 • Server: MPWEB25
HTTP User Agent: CCBot/1.0 (+http://www.commoncrawl.org/bot.html)