Citation linkages

The most common form of quantitative measurement is based upon citation linkages, whereby two documents A and B are linked when document B cites document A by means of a reference to document A in its bibliography. This type of citation linkage is largely understood to be a means of substantiating and building upon previous work, although there are numerous alternative reasons why an author may cite previous work (Case and Higgins, 2000), including criticizing previous work or conforming to social hierarchical mechanisms. Despite these possible alternative reasons for the presence or absence of a citation, the interpretation of a citation-based analysis typically relies on the assumption that citations are provided for positive reasons. Nonetheless, citations do not equate directly to quality; they can be spoken of more correctly as a proxy measure for academic impact.

Counting the citation linkages between documents requires a well-structured citation index. A citation index allows a user to navigate from an article in question, both backwards to previous work and forwards to work which has referenced the article in question. While not the inventor of the concept of citation indexing, Eugene Garfield can be credited with commercializing the process and applying it to the scholarly communication process. Garfield described the concept of his citation index in 1955 (Garfield, 1955), and in 1961 the Institute of Scientific Information (ISI), the company he founded, launched the Science Citation Index. For many decades, the standards in citation indexing were the ISI citation indexes (now part of Thomson Reuters and collectively known as Web of Science). In recent years new citation indexes have emerged such as Scopus (http://www.scopus.com), CiteSeerX (http://citeseerx.ist.psu.edu) and Google Scholar (http://scholar.google.com).

The main differences between Scopus and Web of Science are related to the breadth and depth of coverage (how many journals they cover over how many years). Web of Science, for example, indexes fewer journals but has coverage back to the 1980s for many titles and as far back as 1954 for some; conversely, Scopus indexes more journals, but citation counts are only accurate in content published from 1996 onwards. However, the two indexes operate on the same principles: receiving a defined set of journal articles and their references directly from the publisher, initially as print-copy for scanning and optical character recognition (OCR) and latterly as an e-feed. In this way, Scopus and Web of Science capture and index bibliographic metadata for a selected group of high-quality titles, allowing them to match citing references to target articles with a high degree of accuracy. These metadata and the associated citation counts are then available for downloading in record sets, for analysis by publishers and other stakeholders.

Google Scholar and CiteSeerX operate using an autonomous citation indexing principle, whereby indexing robots look for scholarly material by trawling the Internet, including the major journal content hosted on delivery platforms (such as Wiley Online Library, ScienceDirect, and so on), centralized subject repositories such as arXiv, institutional repositories, or even researchers’ personal web pages. Once the robots have identified suitable content, indexing algorithms extract bibliographic metadata and citing references, and these are matched to existing articles in their database. However, there have been some indications that some issues of quality arise from the automated approach to crawling for metadata (Falagas et al., 2008).

While the above citation indexes aim, to a greater or lesser extent, to cover the full gamut of subject areas in the journals universe, there are also subject-specific citation indexes. Examples of these include the Astrophysical Data Service for astronomy and CitEc for economics. Country-specific indexes also exist, including the Indian Citation Index and the Chinese Social Sciences Citation Index. Journal editors are sometimes quite enthusiastic about these indexes, but as they cover only selective parts of the journals universe, and since they often carry less complete metadata than the ‘universal’ citation indexes, wide-scale journal impact analysis is more difficult using them.

In addition to the standard document B cites document A citation, other forms of citation-based linkages are possible, such as bibliographic coupling (Kessler, 1963) – where two documents cite a common third document (document C) – or co-citation studies (Marshakova, 1973; Small, 1973) which measure the number of times documents A and B are cited together in subsequent literature.

Bibliographic coupling and co-citation studies of the journal literature have until recently been small-scale studies and largely confined to the academic realm of scientometrics because of the need to have access to the underlying raw citation data. However, the same principles can be applied to any collection of documents. In general, such relationships are used to demonstrate similarities between the topics of the articles, rather than as a proxy for impact; for example, SciVal Spotlight, a tool based on Scopus data, identifies potential peers for collaboration using this approach. The advent of institutional and subject-based repositories, which make their metadata readily available for harvesting, means that these measurements are becoming more commonplace.

Subject-specific citation differences

Researchers communicate research outputs of different subject areas in different ways. Some predominantly use journals, while others use a much broader range of media such as monographs, book chapters, working papers, reference works, handbooks and conference proceedings. This has dramatic consequences on the application and validity of journal-based citation analysis in different disciplines.

In Table 11.1, the first column, ‘Importance of journals (%)’, describes the proportion of references within ISI-indexed journals which refer to other journals (Moed, 2005). The data set consists of the references of original articles and reviews published in the ISI indexes during 2002, with only references to items from 1980 onwards being considered. In molecular biology and biochemistry, 96 per cent of references are to other journal articles, whereas in the humanities and arts, only 34 per cent of references are to other journal articles. A citation analysis based on journal citations will, therefore, capture much more of the research communication in molecular biology and biochemistry than it will in the humanities and arts.

The ISI citation indexes provide variable coverage of different subject areas, with some areas being better represented than others. The column headed ‘ISI coverage of journal literature (%)’ in Table 11.1 indicates the proportion of references to articles in journals indexed by ISI, and therefore visible in any citation analysis. The journal coverage is highest in molecular biology and biochemistry and lowest in the humanities and arts.

Multiplying the importance of journals by the journal coverage provides a value for the effective ISI coverage of the totality of research communication in that particular subject area. In the humanities and arts, only 34 per cent of communication is via journal articles and only 50 per cent of those journal articles are indexed in ISI. This gives an overall coverage value of 17 per cent.

This combination of the significance of journal communication and the actual coverage of the journal literature indicates that the questions that can be answered through citation analysis vary greatly between subject areas. Perhaps even more importantly, they will vary between different citation indexes: Web of Science, Scopus, Google Scholar, CiteSeerX, and so on.

Finally, these differences will also vary over time. The data reported in Table 11.1 date from an examination of the ISI citation indexes in 2002. Since then, coverage has increased, most notably with the October 2008 integration of the ISI proceedings indexes. In some cases this has had a substantial effect on the coverage of subject areas. Table 11.2 shows the proportion of selected subject areas comprised by the journal as opposed to the proceedings citation indexes. Those subjects – primarily computer science and engineering – where conference proceedings comprise more than 60 per cent of all content from 2000 to 2013 are shown, along with those where such documents comprise less than 0.5 per cent. Clearly, journal content is a less important vehicle for communicating research in some subjects than in others.

Since 2008, the Conference Proceedings Citation Index has been joined by the Book Citation Index and the Data Citation Index. Thomson Reuters has suggested that 10,000 titles will be added to the Book Citation Index each year and some initial work has been completed (Leydesdorff and Felt, 2012) looking at whether book citation data are robust enough for reliable bibliometric analysis. This may yet become another key focal area of analysis for publishers. However, for now, longitudinal comparisons need to be undertaken with the knowledge that coverage is not fixed, but is constantly in motion.

Citation distributions

The distribution of citations to articles follows a skewed rather than normal distribution, that is a small number of items is exceptionally highly cited, with the majority of items being seldom cited. A study of biomedical journals (Seglen, 1992) reported that: ‘Fifteen per cent of a journal’s articles collect 50 per cent of the citations, and the most cited half of the articles account for nearly 90 per cent of the citations. Awarding the same value to all articles would therefore tend to conceal rather than to bring out differences between the contributing authors.’ Similar findings have been reported in the fields of immunology and surgery (Weale et al., 2004) and for articles published in the journal Nature (Campbell, 2008). Figure 11.1 shows that a similar pattern exists for Australian publications across a range of subjects.

A consequence of this skewed distribution is that average values tend to be highly influenced by the presence or absence of a handful of highly cited items from one year to the next. The implication is that consideration should be given to the type of statistical method or test that is applied when examining sets of citation data. Non-parametric methods will, in many cases, be more appropriate than their parametric counterparts.

The impact factor

The impact factor is a measure of the average number of citations to the articles contained within a given journal. This calculation is based on the citations originating from the journal subset indexed in ISI’s Web of Science, comprising approximately 12,000 journals. In practice, only those journals which appear in either the science or social science citation indexes will receive an impact factor; this represented 10,647 titles in the 2011 JCR.

The impact factor imposes some special restrictions on the calculation of this average, limiting it to citations during a defined period of time, to articles published in a second defined period of time. The definition of the 2012 impact factor is provided in the equation below:

$2012\mathrm{impact}\mathrm{factor}=\frac{\mathrm{Citations}\mathrm{in}2012\mathrm{to}\mathrm{all}\mathrm{items}\mathrm{from}2010\mathrm{and}2011}{\mathrm{Number}\mathrm{of}‘\mathrm{citable}’\mathrm{items}\mathrm{from}2010\mathrm{and}2011}$

The impact factor was created by Garfield in the early 1960s as a measure to select new journals to add to his growing Science Citation Index. By aggregating author-level citation data to the level of the journal, he could determine which journals were most commonly cited. For a fuller historical explanation of the origins of citation indexing and the impact factor, see Bensman (2007). Journals which Garfield identified as heavily cited but not indexed at that time were then added to the citation index. Garfield noted that a relatively small core of journals was responsible for the majority of citations, and this allowed him to cost-effectively cover a large proportion of cited articles, without necessarily indexing the entire corpus of research. In an essay currently available on the Thomson Reuters website this inner core of journals is highlighted:

The suitability of the impact factor as a measure of journal quality has been thoroughly debated (e.g., Cameron, 2005; Seglen, 1997) and we do not propose to revisit old arguments here. Suffice to say, despite correctly noted shortcomings, the impact factor is a powerful metric. It is the preeminent metric in the author, library and research funding community. While many of the following JCR metrics may well not be recognized and are seldom used by journal stakeholders, the impact factor almost always will be. Although it is flawed and frequently used in an inappropriate manner, it cannot be dismissed.

Total citations

While the average citation per article calculation that is the impact factor mitigates for the effects of journal size, it has a tendency to favour review journals which typically publish a relatively small number of highly-cited articles (although see the section ‘Review articles’ on p. 273 of this chapter). The total citations measure provides a broader perspective by counting citations from all citing articles from the current JCR year to all previous articles (from any year) in the journal in question. Such a metric will obviously highlight larger and higher-quality journals. In theory, as the journal’s total historical output, and hence citable material, can only ever increase from year to year, the total citations value should also increase from year to year. In practice, however, obsolescence of the literature occurs (see ‘Cited half-life’ below), with material becoming less likely to be cited the older it becomes. This, to a degree, keeps in check the year-on-year rise of the total citation figure.

Cited half-life

An article’s citation distribution over time can be monitored and characterized. Taking an idealized article, the citations per period will typically rise over time to a maximum. This maximum is dependent on the inherent quality of the article, the subject area (medical sciences will reach a peak more rapidly than social sciences) and the type of document (short communications will typically see an earlier peak than original research articles). From this peak it will then drop off, again at a rate dependent on quality, subject area and the type of article.

The cited half-life value characterizes the age distribution of citations to a journal, giving an indication of the rate of obsolescence. The cited half-life as defined by ISI in its JCR is the median age of the papers that were cited in the current year. For example, a cited half-life in JCR 2012 of 6.0 years means that of the citations received by the journal from all papers published in 2012, half were to papers published between 2007 and 2012.

The desired value for a cited half-life varies depending on the journal. A long cited half-life for a journal can be interpreted as meaning longevity in articles and a journal that serves an archival purpose, whereas a short cited half-life can indicate articles which are on the cutting edge but quickly rendered obsolete by the pace of change in the subject area. Both of these qualities are desirable for different reasons, and it is observed (Tsay and Chen, 2005) that for general and internal medicine and surgery there is no correlation between cited half-life and impact factor.

Immediacy index

The immediacy index is an indication of how rapidly citations to a journal take place, and is calculated as the ratio of the number of citations received in the current year to the number of citable items published in the current year. Within-year citations indicate that the research is being built upon rapidly, which is a desirable outcome for any journal. Contrary to the cited half-life data, it was observed that for general and internal medicine and surgery there is a correlation between immediacy index and impact factor (ibid.).

Subject aggregated data

Since JCR 2003 (published summer 2004), ISI have produced data aggregated to the level of the subject category in addition to the journal-specific values mentioned above. These data allow for benchmarking against peer journals, and enhance the usability of the journal-specific data. The data that is provided at a subject level includes: total citations, median impact factor, aggregate (mean) impact factor, aggregate immediacy index, aggregate citing half-life, aggregate cited half-life, and total articles.

By comparing, for example, the change in a journal’s impact factor over time against the change in the median or aggregate impact factor over the same time period, one can determine whether changes at a journal level are to do with the quality of the articles within the journal itself, or merely a reflection of an overall trend at the subject level. This is particularly useful when explaining step-changes in impact factor across a group of journals.

One note of caution should be sounded when performing longitudinal analysis of the journals in the JCR: the ISI journal universe is constantly in flux. The net change in the number of titles is positive, but journals cease, merge and split, while some are simply dropped as they no longer fulfil the selection criteria as research trends evolve over time. Conversely, new journals, and occasionally new subjects, are added to better reflect the current status of research. This change in the composition of the indexes, and hence the JCR, means that care needs to be taken when interpreting data to ensure that the observed result is a true result, and not simply an artefact of the evolving composition of the index.

Using JCR metrics in promoting journals

In the JCR there exist a number of metrics that can be used to rank journals. The choice of one particular metric over another depends on the message to be conveyed and the ranking of the journal in each of the metrics. For instance, a large journal without a particularly high impact factor may be ranked highly by total citations, and so may adopt that as its USP in marketing messages. That said, the impact factor is still the primary metric to be given weight in the community; indeed, even Elsevier, which produces and promulgates its own alternatives to the impact factor, such as SNIP and SJR, still uses this metric on its journal pages.

Ultimately, however, journal editors and publishers are competing in a crowded world for the attention of readers and potential authors. Marketing messages are prepared and distributed which make use of a variety of metrics, with the intention of appealing to as many potential authors as possible, and with the intention of improving those metrics in the future.

Review articles

Review articles typically attract more citations on average than primary research articles (see, for example, Moed et al., 1996; Peters and Van Raan, 1994). The effect of this can be seen in ISI impact factor listings, with many subject categories topped by review journals.

There is, however, some evidence of journal-specific effects here. Average citation rates of review and research articles in New Phytologist, for example, show marked differences, while the review and regular research articles in Journal of Urology and Alimentary Pharmacology and Therapeutics exhibit less pronounced differences (see Figure 11.3a–c).

One explanation for these differences is the nature of the review articles themselves. Review is a catch-all term for numerous different document types, ranging from a full comprehensive review, to a mini-review, to a perspective, or to a tutorial. This heterogeneity is likely to lead to local differences in expected citation counts. For example, since 1985 New Phytologist has published its Tansley Reviews, reviews written by specialists but aimed at a readership outside that which could be expected from a specialist review journal. This prestige may elevate them further above research articles than might otherwise be expected.

Because of the powerful effect they can have, many journal editors are keen to publish review articles. This is not simply to increase the impact factor however. The data in Figure 11.4 suggest that as well as having a positive effect on the impact factor, review articles are typically downloaded significantly more often than primary research articles and can broaden readership.

It is interesting to note the gross differences in download ratios for the three journals in the different subject areas. Broadly speaking, the ratio for the anatomical science journal more closely mirrors the citation ratios of the medical journals (see Figure 11.3b and 11.3c), while the ecology and microbiology journals reflect the citation ratios of the plant science journal (see Figure 11.3a).

Writing a high-quality, comprehensive review article takes a significant amount of effort on the part of an author; most are more driven to publish primary papers instead because it is these that earn them tenure or continue to advance their reputations. Journal editorial teams also invest a significant amount of energy into devising strategies for acquiring review articles.

The question of whether the proliferation of review articles is desirable and whether this effort is warranted has been investigated in the pathology literature by Ketcham and Crawford (2007). They identified a sixfold increase in review articles between 1991 and 2006, compared with a twofold increase in primary research articles. Similarly, examining papers using hepatitis as a title or key word, the authors identified a 13-fold increase in review articles over a 20-year period, compared with a sixfold increase in primary research articles. In both cases, the growth of the review literature was largely outside review journals. Most importantly, from the perspective of a journal editor who may be seeking to improve an impact factor, the authors demonstrate that only a small proportion of review articles achieve a high number of citations. Therefore, the energy put into acquiring review articles by journal editorial teams may, in many cases, be misdirected. The degree by which review growth outstrips article growth may, in any case, be slowing; a review of the whole of Web of Science in the decade from 2003 to 2012 shows that article counts grew by 51.1 per cent while review counts grew by 86.5 per cent.

Increasing the focus on areas more likely to attract citations

Within all journals some subject areas will be cited more frequently than others, or will fit the two-year impact factor window better. It is not inconceivable that editors might devote more space in a journal to areas more likely to attract citations, although many editors oppose this practice as it would result in their journals not representing full and balanced coverage of their disciplines. Philip Campbell of Nature, quoted in an article in the Chronicle of Higher Education (Monastersky, 2005), rejected the suggestion that Nature focuses on areas likely to attract more citations by stating that if that were the case the journal would not publish papers in areas such as geology or palaeontology and would focus more on subjects such as molecular biology.

Picking up papers from higher-impact journals

In many cases, an author will submit to a high-impact but broad-based journal in preference to one of lower impact but which is perhaps more suited to the subject matter of the manuscript in question. An unofficial hierarchy of broad to niche and high to low quality emerges. If a generalist journal rejects the paper as of insufficient broad interest, but methodologically sound, some high-level, subject-specific journals now informally encourage authors to submit referees’ comments along with their manuscript – this frequently negates the need for further peer review as a decision can be made on the suitability of the paper using the reviews from the first journal. It has the dual effect of increasing the speed of review and publication while lessening the peer review load on the community.

A more organized attempt at transferring reviewed papers between journals has been established in neuroscience. The Neuroscience Peer Review Consortium (http://nprc.incf.org/) has been operating since 2008, with the intention of reducing the load on reviewers. At the time of writing, 40 journals are participating in the initiative. On receiving a rejection, authors are given the option of having the reviews of their article automatically forwarded to another journal of their choice in the consortium.

Reducing the denominator

ISI does not include all document types in the denominator of their calculations of the impact factor (see the equation on p. 268 above), whereas all citations to any document type are counted on the numerator. This can lead to situations where some citations are not offset by the presence of a publication on the denominator, effectively meaning that these citations are ‘free’ citations. What ISI does include it terms as ‘citable’ items, which typically include source items. Source items typically include research articles, review articles, case reports and articles published in any supplements. Items frequently excluded from the denominator of the calculation include letters (unless they function as articles, for example in Nature or Ecology Letters), commentaries, meeting abstracts, book reviews and editorials. The document type designation of a journal’s papers can be readily determined once they have been indexed in Web of Science and some guidance has been published on this topic (McVeigh and Mann, 2009); however, it is not always clear how that decision has been reached, and this can be a source of some frustration for editors and publishers (The PLOS Medicine Editors, 2006). Editors and publishers are able to contact ISI to request that they treat certain article types as non-citable items, but this channel is informal, and it is entirely at ISI’s discretion as to whether requests are granted.

Self-citation

Self-citation, and specifically a journal editor suggesting to authors during the peer-review process that they may wish to consider citing more work from that journal (either suggesting particular papers or suggesting that the authors identify some themselves), is regarded as the least acceptable way for a journal to improve its impact factor.

Editorials focusing on the journal’s own recent content can also be used to increase the number of citations. In 2004 this approach was observed with an editorial citing 100 of its own articles from previous years (Potter et al., 2004); furthermore, the editorial was published in not one but five different journals. In reality, only one of these editorials was indexed by ISI, and these self-citations were actually only a small proportion of the citations received and did not unduly disturb the journals’ impact factor. While obvious effects such as this can be readily identified (Reedijk and Moed, 2008), more subtle effects prove harder to quantify, although with the widespread availability of citation data it is unlikely that these practices could go undetected indefinitely.

The JCR now identifies the proportion of self-cites for individual journals, and it is possible to re-calculate an impact factor once the effect of self-citing has been removed. In JCR 2011, published in summer 2012, the impact factors of 50 journals were suppressed and not reported, due to excessive self-citation. A notice accompanying JCR 2011 reported:

While the language of this note has been altered from previous years, which made explicit reference to self-citation (Journal Citation Report Notices, 2008), it is worth noting that the number of suppressed titles has increased from eight in 2008 to 50 in 2011; instances of manipulation, the success rate of detection, or both, are increasing.

However, it is important to acknowledge that there is often a good reason for a journal having a high self-citation rate. Journals which are very specialized or which are the dominant publication for a particular subset of a subject are more likely to show high rates of self-citation than those where research is spread among a number of journals with similar aims and scope.

A new, and subtler, form of self-citation appears to have emerged in recent years, whereby members of a cartel of journals cite each other in much the same way as the editorial described above. As it is not the journal benefiting that is increasing citation to itself, but rather one of a multitude of other titles that cite it, such gaming can be difficult to detect. One example of this was identified recently (Davis, 2012) and led to the journals involved being stripped of their 2011 impact factors.

h-index and descendents

Before describing some of the new methods for ranking journals, it is worth mentioning in passing the emerging methods for ranking individuals based not on the aggregate performance of the journals in which they have published but the actual citation performance of their individual articles. Beginning with Hirsh’s h-index (Hirsch, 2005) – and quickly followed by a number of related indexes including the H¹ index (Kosmulski, 2006), the g-index (Egghe, 2006) and the R- and AR-Indexes (Jin et al., 2007) – there has been a growing realization that the journal as the unit of measurement is not the most appropriate measure.

Hirsch’s original h-index as applied to the individual is calculated as the natural number ‘h’ such that the individual has published ‘h’ articles which have each been cited ‘h’ or more times. An h-index of 3, therefore, means three papers have been cited at least three times each, while an h-index of 10 means ten papers have been cited at least ten times each, and so on.

Criticisms have been levelled against the h-index since its inception (Costas and Bordons, 2007; Schubert and Glänzel, 2007) and the subsequent indexes attempt to address these criticisms while retaining the simplicity of the original index. Some of these prove better suited to the task than others (Bornmann et al., 2008).

Impact factor modifications

Numerous alternative impact factor measurements have been proposed over the years. Examples of these include: the ‘per document type impact factor’ (Moed and Van Leeuwen, 1995), where the differences in inherent citability of different document types published within the same journal – for example, original research articles, review articles, letters – are mitigated; the ‘rank-normalized impact factor’ (Pudovkin and Garfield, 2004), where a percentile ranking based on the impact factor of all the journals in a particular subject category is calculated; and the ‘cited halflife impact factor’ (Sombatsompop et al., 2004), where the age of the cited references is factored into the calculation.

The fact that none of these has gained widespread acceptance may be interpreted in a number of ways. It may be simply a consequence of a ‘better the devil you know’ attitude in the community. While the impact factor is imperfect, who is to say that any new measurement will be more equitable? Alternatively, it may be a deeper-seated dissatisfaction with the whole process of ranking by citations. Another explanation is related to the fact that all these modifications rely upon the underlying citation data as provided by ISI – which may have little incentive to change a formula which works perfectly well within the defined limitations that it has set out.

Alternative journal indicators

In traditional journal ranking measurements such as the impact factor, a citation from a high-impact factor journal is treated in exactly the same way as a citation from a low-impact factor journal, i.e. no account is taken of the citing source, only that a citation linkage exists. In an eigenvector-based journal measurement, such as the Eigenfactor (http://eigenfactor.org) or the SJR indicator (http://www.scimagojr.com), the computation takes into account a quality ‘characteristic’ of the citing journal.

It should be noted that the eigenvector style analysis applied to the ranking of scholarly journals is not a new phenomenon; indeed the process was described and applied to a selection of physics journals in the mid-1970s (Pinski and Narin, 1976). The re-emergence of this type of measurement has been driven by the success of the Google PageRank algorithm, which is itself based on eigenvector analysis. Google defines PageRank as follows:

The most significant of the eigenvector style measurements is the SJR Indicator, which was released in November 2007. Its significance is not related to the mathematics of the calculation but to the underlying source of the citation data, which in this case was not ISI data but Scopus data.

Also based on Scopus data but taking a different approach is the SNIP (www.journalmetrics.com). It is the first metric to be calculated for the whole journal list that seeks to correct for the differences in average citation impact among different subject areas, to allow the comparison of journals across subjects. An average citations-per-paper is calculated for a journal and then divided by the ‘relative database citation potential’ (RDCP), which measures how likely it is that the journal should be cited, given how many citations are made overall by articles in the journals that cite it.

Comparing different ranking systems

The differences between the journal rankings as produced by three different ranking systems – JCR, Eigenfactor and SJR – can be examined, and relative performance in each scheme determined. The analysis in Tables 11.3 to 11.7 examines the rankings of journals in dentistry and associated subject areas.² Note that it is currently not very simple to perform a like-for-like comparison, nor even to establish what the most appropriate comparison would be.

Of the 145 unique titles covered in these systems, 59 are common to all three systems. Tables 11.4 to 11.7 describe the top ten ranked titles in terms of their impact factor, Eigenfactor, SJR and SNIP respectively. A superficial comparison of this small sample of data suggests that there are differences between the three systems. Many journals highly ranked by impact factor have relatively low rankings by Eigenfactor and vice versa. This is perhaps unsurprising: the impact factor takes account of the number of articles in a journal, while the Eigenfactor is closer to a measure of total citation in that it does not.

Similarly, when looking at the SJR and SNIP rankings (Tables 11.6 and 11.7) there are few journals which do well in all four systems. Only the journals Dental Materials and Journal of Clinical Peridontology are ranked in the top ten by all four metrics. Even the Journal of Dental Research, which is generally acknowledged to be a leading journal in the field, does not crack the top ten for the SNIP and only just does so for the SJR. Whatever it is that each metric is measuring, it appears not to be the same thing, and even using a number of metrics in parallel will struggle to separate all but the top tier of journals.

COUNTER Journal Report 1 and cost per access

From Journal Report 1, library administrators can compare the level of full-text downloads of their journal collection over monthly reporting periods. A common measure that is then derived is the cost per access (cost per use), which enables a comparison of the cost-effectiveness of different parts of a collection. Although superficially a simple process (to divide the cost of the journal by the number of full-text downloads in a specified period), the range of mechanisms in which journals are sold to institutions has a large bearing on the relevance and validity of this figure.

In the print era, journals were typically sold as single entities. Subscription agents made the process of subscribing to different journals from the same publisher a much simpler proposition, but, in essence, the cover price of a journal was what was paid, and a publisher’s revenues could be estimated by multiplying the cover price (minus an agent’s commission) by the number of subscriptions. In today’s online era, journals are sold as a mixture of single sales and larger bundles, and to a combination of individuals, institutes and vast library consortia occasionally spanning an entire nation, where the terms and conditions of the deal are collectively brokered. A consequence of this is that access to the journal literature has never been greater, with many institutions subscribing to a publisher’s entire collection of titles.

A by-product of this bundling process is that the actual cost to the library of an individual title is typically significantly less than the cover price. Precisely how much less will vary depending on the particular subscription model the publisher operates. Now factor in the changes in the operational cost structure of the library as a result of print to online migration (Schonfeld et al., 2004) and a simple cost per access calculation suddenly becomes a far more intricate undertaking (Price, 2007).

Usage factor

As with counting citations, the number of downloads is determined by the number of articles online and accessible. All things being equal, a larger journal will experience more downloads than a smaller journal. This size effect can be mitigated by calculating an average number of downloads per article, in the same way as the impact factor is an average of citations per article. This so-called usage factor, as proposed by COUNTER and the UK Serials Group (UKSG) (Journal Usage Factor, n.d.), has the potential to make meaningful comparisons between journals based on their usage, although there are a number of problems which will need to be overcome before such a measure can be universally accepted.

The main challenge is in creating a measurement which is resilient to deliberate and systematic abuse. This is of particular relevance when considering a usage-based pricing model, or where reward of individuals is based (even partly) upon the usage of their journal articles by others. The project’s Stage 2 analysis (Journal Usage Factor, 2011) concluded that the chances of such manipulation being conducted successfully were small, but there exist a small number of scenarios in which the metric could be successfully gamed.

COUNTER’s planned response to this seems to be twofold: to ensure that definitions of downloads and the exclusion of suspicious activity are sufficiently precise to minimize the influence of deliberate manipulation; and to apply statistical audits to publisher usage data, to check for signs that gaming has taken place. Whether these efforts will be successful remains to be seen.

European Reference Index for the Humanities

An example of peer opinion being used to rank journals is the European Reference Index for the Humanities (ERIH), a project run by the European Science Foundation (ESF) (http://www.esf.org). The ERIH project provides categorized lists of journals in 14 areas of the humanities. Expert panels split the journals into three categories, A, B and C, based on peer review. However, ERIH reports that this was misinterpreted in the community as being qualitatively hierarchical (ERIH Foreword, n.d.) and that the differences between the categories were of kind, not quality. It should be noted, however, that in a document no longer available (ERIH Summary Guidelines, n.d.) both qualitative and categorical criteria appeared to be applied to A, B and C classifications.

This decision may have been influenced by debates on the margins between the different categories, and of the overall wisdom of applying any such ranking to the humanities. In a joint editorial entitled ‘Journals under threat: a joint response from history of science, technology and medicine editors’, and archived online in numerous discussion lists and forums, editors from over 40 journals raised concerns about the process:

Instead of an A, B and C classification, journals have been divided into INT(ernational)1, INT(ernational)2, NATional and W categories. The difference between INT1 and INT2 seems to be partially qualitative. As at the time of writing, this has been completed for 12 of the 14 subjects, with the results hosted by the ESF.² The stated motivation for creating the ERIH was to address the problem of low visibility for European humanities research, in that:

It remains to be seen what the long-term future of the ERIH will be.

Other efforts to rank journals using qualitative methods have encountered complications. An attempt by the 2010 Excellence in Research for Australia (ERA) exercise to rate all journals from A*–C using expert review caused immense controversy (Creagh, 2011). The rankings were retired for the most recent ERA in 2012.

The UK Research Excellence Framework

Following the retirement of the UK Research Assessment Exercise (RAE) in 2008, the Higher Education Funding Council for England (HEFCE) began planning its successor, the Research Excellence Framework (REF). A November 2007 HEFCE consultation document (Research Excellence Framework, 2007) noted that quantitative indicators, and particularly bibliometrics, will be a key element of the judgement of the quality of research. One of the major criticisms of the RAE – that it compelled authors and institutions to chase high impact factor journals – is explicitly dealt with in the consultation paper; HEFCE stresses that the bibliometrics applied will not involve the use or recording of journal impact factors.

The current plans for the incorporation of bibliometrics into the 2014 REF are based on a pilot study conducted in 2008–9 (Bibliometrics Pilot Exercise, n.d.). A fuller explanation of the metrics to be applied (Bibliometrics and the Research Excellence Framework, n.d.) indicates that a normalized citation measure will be applied; this is a fairly robust metric in which the actual number of citations received is divided by the average received by other publications in the same subject in the same year, to give a measure relative to a world average of 1.00. If the result is 1.25, the article is cited 25 per cent more than the average; if 0.75, it is cited 25 per cent less. Each researcher will be invited to submit up to four publications for consideration; therefore the REF will not look at the entire corpus of an institution’s research output (Assessment Framework and Guidelines for Submission, 2012). This measure of relative citation impact will then be used to construct a citation profile, where the proportion of work far below, below, around, above and far above the world average will be charted. In this respect, it is very similar to the ERA exercise that preceded it in Australia (see below).

Currently, it is not clear how granular the subject scheme will be, but it is likely to be based on the Scopus database, for which world average citations received in a subject area (often called ‘citation baselines’) can be calculated but are not publicly available. Scopus was selected as the data provider for the REF, as it was for the ERA, possibly on the basis that it covers more publications.

An interesting question that arises both from the RAE and the REF is to what extent researchers modify their behaviour in response to the evaluation process itself. In an assessment of UK science spanning the period 1985–2003, Moed (2008) concluded that the observed behaviour of UK scientists varied depending on the assessment criteria in the prevailing RAE. For instance, in RAE 1992 when total publication counts were requested (rather than the current situation of submitting a subset of ‘best’ work), UK researchers dramatically increased their article output. Furthermore, in RAE 1996 when the emphasis shifted from quantity to quality of output, the proportion of papers from UK researchers in high-impact journals increased. When a system of evaluation is created, generally those who are being evaluated will rapidly work out the practices to adopt in order to allow them to exploit the evaluation criteria. This ‘gaming the system’ is an inevitable consequence, and is an important factor to consider when developing any evaluation framework.

Excellence in Research for Australia (ERA)

Another example of a ranking for journals can be found within the ERA initiative (Excellence in Research for Australia, 2008), announced in February 2008. ERA aims to assess research quality of the Australian higher education sector biennially, based on peer-review assessment of a number of performance measures, including bibliometric indicators. The first ERA was conducted in 2010 and reported in 2011; the following ERA was conducted in 2012 and reported in early 2013.

While more traditional peer-review elements such as income generated and various esteem measures (such as specific awards and memberships of respected bodies) will be considered by the peer-review panels, so too will be publication outputs and their citation impact, with the results rating a university with a unit from 1–5 (ERA, 2012). The outputs will only be counted if they appear on the ERA journal list (which, as noted before, no longer carries the A*–C ranking). The method for calculating the citation impact of an institution’s publications is similar to that to be employed by the REF – actual citations received divided by a baseline for publications in the same year and subject – except that both a global and a national baseline are used.

Three types of analysis are employed: the distribution of papers based on world and Australian citation centile thresholds; average relative citation impact; and, as with the REF, the distribution of papers against relative citation impact classes. The centiles are calculated such that the most cited papers in a subject are in the 1st centile and those receiving no citations are in the 100th centile.

Only those publications listing an author based at the institution at the date of the census will be considered by the panel. This permits a degree of gaming, with institutions ‘poaching’ well-cited authors prior to the assessment date. Gaming is also possible in the subject scheme applied: this uses a four-digit Field of Research code, dividing up science, social science, arts and humanities into 164 subjects (although arts and humanities subjects will not have citation indicators applied), and a single paper can be allocated to more than one subject. It is up to the submitting institution to decide the ‘share’ of each subject for a given paper, meaning that shares of less cited papers could be allocated to a less crucial subject, and shares of more cited papers allocated to subjects of preference.

Despite this, and despite ongoing debates about the influence of self-citation, ERA was the first national research assessment scheme to successfully and consistently apply robust bibliometric indicators. There is no indication that this will change in the immediate future.

Combining usage and citation statistics

It is often argued that local rankings are more appropriate than global rankings for a librarian managing his or her collection, and many advocate the counting of local citations (i.e., citations from and to articles authored by local authors) rather than all citations as a more valid measure of how useful that particular journal is to the local authors.

Currently, the reporting of full-text downloads takes place at the level of the subscribing institution, and allows administrators to observe usage of their content at the local level. In terms of ranking journals, however, this will produce a localized ranking based on the specific characteristics of the subscribing institution in question. These characteristics – such as whether the institution has a large undergraduate programme or is purely research focused, or whether the institution has a broad or niche subject base – play an enormous role in determining the download figures, and hence the ranking.

In early 2006, ISI announced a product which would allow the combination of COUNTER-compliant usage data and institution-specific publication and citation data. The Journal Use Report (Newman, 2005) promised to ‘provide users with a 360° view of how journals are being used at their institution’. With emerging standards such as SUSHI (Standardized Usage Statistics Harvesting Initiative, n.d.) enabling the aggregation of usage data from different publishers, the activity of evaluating journals from a local standpoint becomes a far simpler proposition than previously. The Web of Knowledge Usage Reporting System (WURS) is now available (http://wokinfo.com/usage/), but this only reports usage from the subscribing institution, unlike the citations reported, which are counted from any citing source. While this allows an institution to establish the usage of subscribed materials (which many do anyway) it is unable to give a global view of usage across all subscribers.