On the usefulness of Citations in Academic Evaluation

 This article was actually written in March 2011. I just got around to "sounding off" about it now ! 

There is  increasing recognition by  governments world over  that  the vitality and excellence of   a  nation's  academic    and research  system  plays a central and   determining role in all aspects of  National  well being. It particularly   affects the ability of   nations to compete effectively in the globalized economic arena of  an ever  more knowledge driven world economy.   Increasing resources are being allocated to fund higher  education and research  and  there is a concomitant   increase in  anxiety - both among the governors and the governed - regarding the emplacement of  mechanisms of evaluation that will ensure that the resources invested are not feed for white elephants or polish for ivory towers  but rather select real talent,  nurture real  excellence , reward real hard work  and inspire real striving  : which will lead to  original contributions to the growth of  basic knowledge,   the development of  technology and the promotion of  human welfare  and culture in society at  large.  In this context the increasing use of Citation based measures of individual and institutional performance in all developed scientific systems  challenges  the Indian academic system’s  continued reliance on largely subjective or formal  indices of academic and scientific worth  coupled with an often  decried mediocrity in motivation,   achievement and innovation.  This challenge is particularly relevant  at this  conjuncture when the system needs to be reformed to evade the stultifying effects of entrenched networks of  patronage and mutual reward  of academic  and scientific elites . The recent vogue of publication of Global University rankings  (by  a multitude of agencies ranging from the Times of London to the University of Western Australia  or Shanghai ) which use Citation performance of University faculty and scholars as an important component of the  ranking method has also brought the issue to the front page of public perception.

Starting from the chronic underdevelopment of a colonial society,  India has made much progress  in  providing cohorts of  scientific and technical   staff who can contribute well when transplanted  to the   scientific ecosystems of  the developed world.  However the achievement of scientific excellence within the country, specially as regards  advances in fundamental science – as opposed to   localization of established scientific techniques and procedures - has been much more modest.  A considerable component  of the obstacles to   emergence  from scientific underdevelopment, backwardness,   and mediocrity  may well be the  fact that  hitherto the evaluation of  scientific merit of individuals and groups  has largely been left to the subjective evaluation   of the very elites whose mediocrity is  lamented . It has long been the lament of émigré  achievers that it was the certainty of being frustrated by brazenly unfair and opaque  under valuation of their talents that drove them to quit the country.     By the very nature of  science,  competence to evaluate scientific  work is generally  restricted  to   co-workers in the subfield ( for example   Particle  Physics is a sub field of Physics , which is itself divided into  practically autonomous  subdivisions, each with it’s own  standard working `lore’ ,  distinct sociology and  accepted  norms of performance and networking.  There are  divisions such as  Experimental High Energy Physics,  Phenomenology , Lattice Gauge Theory , and String Theory, and it is a rare researcher in one area who is competent to judge work in another area .  Since the density of workers in any sub-field  is rather low  (typically of the order of  1-3  individuals for any division actually present  in an academic Department) ,  the total  pool  of competent   evaluators in the country as a whole is typically less than a 100 individuals.   Naturally this very limited pool of  individuals are linked together by various professional ties   and transactions  as well as mutual interest in fostering professional development including placement in  jobs, placement of  students,  evaluation of theses,  promotions,  recognition  for awards and  so on .    The  restriction of   evaluation  to the subjective opinion of that  very set  of  national senior scientists  and academics, whose stagnation,  imitative repetitiveness and  mediocrity of achievement  the  Indian scientific system needs to bootstrap itself  away from,  functions as a  Catch-22  that    retards   progress.     It is an axiom of scientific administration that   mediocrity perpetuates  itself   by recognizing only the mediocre  while notionally genuflecting to distant  and  irrelevant eminences (the  ``scientists in America have shown syndrome’’ !).  

 There is thus a need   to by pass this obstacle  by introducing   objective   factors of evaluation that cannot be easily faked by a consensus of  the  mediocre,   already in positions of control or eminence,   whose deficient evaluation is difficult to  successfully challenge without recourse to indices that transcend  or bypass the local stagnation and underperformance.   If introduced , such mechanisms  will bias the  entire system positively toward the recognition of talent and performance  and lead to the  evolution of a pool  of actual (as opposed to notional)  talents.  In short there is an urgent and inescapable need for  objective and quantifiable inputs to the performance evaluation process  that can escape  the dangers of nepotism and patronage  influenced evaluation.  

  The need for  indices of achievement that are harder to fake by collusion amongst local/national  evaluators can  now be partially met by taking recourse to   statistical measures of scientific achievement  such as citations  of research workers in research publications  by co-workers in the same   field of research  and various derived measures based on such citations.  Of course the mere use of such indicators cannot  replace,  but only supplement, the  traditional indicators of talent and performance .  Application of mind –in  good faith !-  to the entire available range of information about performance   is - and will continue to be-  necessary to succeed  in attributing   merit    and rewards  in a way that will in the long run lead to the  development of excellence by global standards . Thus   one should  always keep in mind the caveats that: (i)    Evaluation    of  performance and potential can never become purely objective.(ii)  Nor can it escape the need for a the presumption of goodwill  and faith on the part of the evaluators. (iii)  Nor can the same yardstick and benchmarks  be applied uncritically across different subfields  and specializations.  If these reservations – and   additional  modulations  required in the light of subsequent experience when such metrics are used in practice -  are kept in mind then consistent  input  of citation information can     usefully supplement and improve  evaluation .

   For reasons very similar to those adduced above in the National context ,  and in spite  of well functioning and innovative scientific establishments , there is also Internationally a  drive towards greater reliance on   innovative  statistical   measures  designed  to identify  and rank  academics and   researchers according to the  depth  and range of their impact on society and  peers.   The increasing ability to amass and sort vast amounts of information using the Internet  has led to an qualitative change  in actual and perceived role of   bibliographic metrics for the evaluation of   academic performance.  Of course  citation databases  allow  one to rapidly trace the evolution of a scientific idea.  In addition,    researchers in most branches of science now routinely use  Citation Counts, H-index,  Author Rank(based on the Google  web page ranking algorithm !),  Impact Factors  and  so on to rapidly    evaluate  the scientific output of candidates and peers. We shall briefly review some of these terms below and comment on their significance.

  The citation trail that an individual 's scientific  publications generate is      accepted as a means of ascertaining the importance and relevance of the work done.   While scientists as individuals  may be prey to all the human frailties, nevertheless as  a group  they are committed to an “inter-subjective” conception of     truth : the quest for which motivates and drives their work.   The “truth'' in any subfield'' is arrived at by a dialectical process of  discussion and criticism in the format of the scientific journals which  generally  require    that  the prior history of study and research be adequately summarized and referred to as a precondition for publication.    Unjustified claims of scientific seminality or preeminence are always vigorously contested not only by those directly affected but also by  other knowledgeable individuals in the sub-field. Thus  there is an an internal policing mechanism within each  scientific sub-field   to prevent  pretensions and plagiarism   and thus award citation levels justified by actual merit. Moreover no geographical area is now  so dominant as to get away with     recognizing  only its own contributions.  Any sensible and genuine scientific worker knows he has little possibility to obtain   recognition for innovation or study unless he clearly identifies and distinguishes the work done prior to him .    Thus even  if they are reluctant to cite the works of others, specially  competitors (!) ,  there is really no alternative to doing so if they have scientific priority and precedence.   In this way tracking the citation counts and connections of an individual or group is generally an  effective way of tracking their actual impact on science.  

The  inter-subjectivity of   the scientific  process has long been consciously appreciated by scientists.  However   it is only over the   last half century  that rigorous bibliographic research  has established and conceptualized   how science progresses through (and  therefore  can be tracked by tracing)  the  linked percolation process  connecting  waves of scientific   papers that inspire succeeding papers that cite the papers that inspired them.  Such  research   has clearly established the “  inherent topological and graphical nature of the worldwide citation network”  as   a  crucial property of    scientific  discourse :  as proposed by R. Garner in 1965.   

“The use of citation counts to rank journals was a technique used in the early part of the nineteenth century but the systematic ongoing measurement of these counts for scientific journals was initiated by Eugene Garfield at the Institute for Scientific Information who also pioneered the use of these counts to rank authors and papers. In a landmark paper of 1965 he and Irving Sher showed the correlation between citation frequency and eminence in demonstrating that Nobel Prize winners published five times the average number of papers while their work was cited 30 to 50 times the average.”[Wikipedia]

 Since then multiple studies have re-confirmed the strong correlation between citation frequencies and scientific originality, relevance and influence at all levels of the achievement pyramid.  Before the advent of the digital revolution and the internet,  tracking bibliographic information  and making it available for reference by  scientists or scholars was a complex and almost prohibitive enterprise.  However the Information  Revolution has brought with it easy access to large bibliographic data bases that are now essential to almost every scientist in the internet era.   Some of the well known data bases are the  general purpose Science Citation Index  from Thomson Reuters available online through the Web of Science  database   and the SCOPUS data base available from  Elsevier as a part of the ScienceDirect  database .  Various specialized data bases such as the  Neuroscience Citation Index from  Thomson Reuters,   and the path breaking  and user charge free Stanford Public  Information Retrieval  System  for  High Energy Physics (HEP-SPIRES) are also commonly now  used via the internet thousands of times daily.

  I   illustrate my comments by reference  to HEP-SPIRES since it is available freely on the internet (http://www.slac.stanford.edu/spires/  and at many other ``mirror’’ sites maintained at various high energy physics  research centers across the globe ), is complete for it’s subfield (over a million records updated daily ) and simple to use and understand.  Similar databases are now available for Astrophysics (NASA), Chaos (Maryland), Computing(Computing reviews) ,  Mathematics(MathScinet,  Zentralblatt  MATH), Nuclear Physics (Nuclear Science References W³ Retrieval System )  and so on.  Commercially available data bases  such as the Web of Science  and Science Direct   offer many more `bells and whistles’’ on the basic   functionality of SPIRES .  A successor system (INSPIRE ) has begun to offer many of these  more complex bibliographical metrics.

As an   example  we give in Table 1  the results of a search of the HEP-SPIRES/INSPIRE    database  for   renowned Indian Theoretical High   Physicist Asoke Sen’s bibliographic data.  The search instantly   yields a summary page that shows just how high the  performance bar can be objectively  set even for scientists working within the country.  The summary reveals an extraordinary level of scientific productivity, excellence  and information. Over 212  scientific papers of which 202  are  papers published in recognized (by HEP-SPIRES criteria) scientific  media  and have been cited over 18000 times by various  scientific articles.     With an average number of citations per paper of 86  and an extraordinary  number of   over 100  ``TOPCITE’’  papers ( two with more than 500 but less than 700 citations , 19 with 250-499   citations ,  30 with 100-249 citations and so on)  the impressive citation counts are supplemented by his extraordinarily  high  H(for Hirsch)-index of  75.   The H-index is defined as the maximal number  H  of papers with H or more citations each  for the author in question. Thus  this author has  at 75 , but not 76, papers with 75 or more citations each.  It is known that the total number of citations  $N$ scales with the square of the h-index (with a coefficient of   3.5-4) .

Citation summary results	All papers	Published only
Total number of citable papers analyzed:	212	202
Total number of citations:	18,501	17,534
Average citations per paper:	87.3	86.8
Breakdown of papers by citations:
Renowned papers (500+)	2	2
Famous papers (250-499)	19	18
Very well-known papers (100-249)	32	31
Well-known papers (50-99)	48	45
Known papers (10-49)	86	82
Less known papers (1-9)	22	21
Unknown papers (0)	3	3
Additional Citation Metrics
h-index	75	73

Table 1 :  Results of a search for citation data of renowned Indian String Theorist Asoke Sen on March 13, 2011.

It is interesting to contrast this record of  a largely Mathematically oriented physicist with that of the three giants who defined the Standard Model of Particle physics : Steven Weinberg (184 papers, 49012 citations, H-index 94, 114 topcites, 1 paper cited more than 7000 times one more than 2000 times and 7 more than 1000 times),  Abdus Salam (232 papers, 15679 citations, H-index 53, 58 topcites, 1 paper cited more than 3000 times  and one   more than 1000 times) and Sheldon Glashow  (150 papers, 26921 citations, H-index 59, 65 topcites, 2 papers cited more than 4000 times  and one each more than 3000, 2000 and 1000 times respectively).    At least to those knowledgeable  about these subjects the  citation levels and details   contain a wealth of information on the contributions   and stature of these scientists and the sociology of their fields of research. 

A use of the  freely available SPIRES database to study the citation levels of various other more or less eminent Indian  High Energy Physicists  is also most instructive. One finds   instances   of eminences,  members of various National Academies   and Directors of National institutes who   have accumulated only a few hundred citations in a long career spanning forty years yet habitually sit in exacting judgment on scientists 30 years their junior who already have better recognized work  as far as citation levels go!  Like the dynamic effects of the recently introduced  Right to Information  laws   on entrenched bureaucracies and patronage networks ,  citation  information   has the potential  to upset many comfortable academic apple carts  of the  powerful but often comatose  guardians of entrenched mediocrity in our academic and research system .

The HEP-SPIRES data base allows various methods for further refining the  metrics presented to remove doubts whether the citation record  presents a true picture of the esteem in which the  work done by the author is actually held by his scientific peers.    For instance  one may subtract  self citations  to  check whether the citation picture changes drastically thereby.  Another   check  that can prove  very revealing is the number of cites divided by the number of authors.  In fields such as Experimental  High Energy Physics where teams of thousands of individuals labour on an experiment and publish collectively  it often happens that the papers of the collaboration are cited thousands of times. Similarly when a working group of tens or hundreds of theorists publishes a ``white paper’’ summarizing a consensus in an evolving field it often receives a large number of citations. In such situations a re-normalization  is called for in order.  For example  a division by the number of authors  can however yield a figure of only a few citations per author ! Clearly the true picture regarding the scientific contribution of the individual scientists  in the huge collaborations must be sought by other means than a simple citation count.  In fact there exist methods in such fields for obtaining a true picture of an individuals talent and contribution by studying the  Collaboration Notes contributed by small sub-groups and their citation by the final published paper,  opinions of group leaders  etc.  This example underlines the dangers of uninformed  use of citation numbers,  specially when making comparisons between different subjects.  Thus the use of citation data requires an informed  formulation of appropriate norms for citation performance in each field separately.  The reader who repeats  the above  exercise (performed  using the INSPIRE database  at http://inspirebeta.net/search) will also soon  realize that the automated databases are bedeviled by familiar digital bugbears : the need to discriminate between individuals carrying closely similar names or initials.  There is thus also a need to provide  unique identity labels to authors which has not been addressed satisfactorily so far. 

Another source of confusion is the  use of  Journal Impact Factors  to judge the excellence of  individuals .  Some  administrators have begun to reward and recognize faculty on the basis of Cumulative  (Journal) Impact Factors.  To appreciate the absurdity   and negative impact of such a policy one should first be clear about what   Impact Factors  actually refer to.  Unfortunately many Indian  academics have only a hazy   idea of the meaning of this terms, or  perhaps  actually support the   introduction  of this policy for  even less admirable reasons.  The Wikipedia defines the Impact Factor(IF)  as follows :

"In a given year, the impact factor of a journal is the average number of citations to those papers that were published during the two preceding year^. For example, the 2003 impact factor of a journal would be calculated as follows: A = the number of times articles published in 2001 and 2002 were cited by indexed journals during 2003. B = the total number of "citable items" published in 2001 and 2002. ("Citable items" are usually articles, reviews, proceedings, or notes; not editorials or Letters-to-the-Editor.) .2003 impact factor = A/B”

It also succinctly describes the reasons for the absurdity of using IF to judge individuals and how this fact is also widely appreciated by administrators in advanced countries :

"The impact factor is often misused to evaluate the importance of an individual publication or evaluate an individual researcher. This does not work well since a small number of publications are cited much more than the majority - for example, about 90% of Nature's 2004 impact factor was based on only a quarter of its publications, and thus the importance of any one publication will be different from, and in most cases less than, the overall number. The impact factor, however, averages over all articles and thus underestimates the citations of the most cited articles while exaggerating the number of citations of the majority of articles. Consequently, the Higher Education Funding Council for England was urged by the House of Commons Science and Technology Select Committee   to remind Research Assessment Exercise panels that they are obliged to assess the quality of the content of individual articles, not the reputation of the journal in which they are published."

 Note that  a policy that spuriously uses an index appropriate for journals to judge individuals is not only unjust but also  is   clearly  anti-National inasmuch it seeks to introduce a bias that will  tend to prevent Indian researchers from publishing in national journals such as Pramana     since these do  not have  anywhere near the same IF as many other journals.  Even a person who  performs work  that is  well cited (even though published in a low IF journal)  will not be recognized under such policies.   Indeed over time all sub-dominant If journals would be driven towards a fixed point of low or zero Impact Factor !  The logic behind such thinking seems to be  that of an outsider  who recounts the instances when he manages to penetrate an exclusive club  as a proof of his social merit : even if the club itself may rue the lowering of it’s tone(i.e of  it’s IF !)  by his entry (publication in that journal) ! Like all reasoning  based on snobbery, whether direct or inverted,  it cannot lead to healthy development or behaviour  and specially will not encourage individuals to  mount the challenges to orthodoxy and dominance that are the heart of  good science. Rather it is a genuflection to the established  scientific dominance  order which    countries like India need to challenge if they are ever to come into their own.

In spite of these facts being   well known and  critiques being easily accessible,  administrators in the Indian context continue to use IF and CIF as measures of individual performance.   It also bears mention that the detailed evaluation   criteria published by the UGC as a part of the revision of Pay scales under the 6th Pay Commission  make  no mention of  Citation levels as a basis for judgment of  research performance .   Little interest is evinced by teachers and researchers in Universities and National Institutions towards a reform  of the evaluation system to include citation data to provide a more accurate and objective picture  of individual performance.  Nor has the manifest perversity of using IF and CIF as opposed to more appropriate measures like  cumulative citation counts and  H-index, average citations per paper etc elicited much protest .  Indeed one gathers the impression that  many Indian academics are  motivated to cast doubts  on the citation based evaluation methodologies  using   arguments based on real or imagined anomalies that may occur  and thus  argue against even the  introduction of the use of citation data. It is amusing, however, that often the very same critics of citation based  evaluation are capable of waxing eloquent upon the influence of their own papers when they gain even a modicum of citations.  We have already mentioned that various caveats are in order   when using citation data  but it is our firm belief that  the use of citation data is essential and inevitable at least for those sectors of the  system that have performance levels that are not so low as to escape evaluation on the basis of citations completely !   India can ill afford an ostrich like attitude towards  such a positive  and enabling technological development which has the potential to debunk and dis-empower entrenched but dysfunctional elites and  turbo-charge our evolution as  a Global  scientific power.  Introduction and publication of citation data relevant to individuals selected for any scientific or academic position(and their unsuccessful competitors !)  would go a long way in reforming the academic evaluation system . Such an information  and   driven  reform  would be consonant with the spirit of the age as already seen in the radical effects of the Right to Information and Asset Disclosure  laws which have already opened the working of the Bureaucracy , Legislature and Judiciary to public oversight and reform and induced a welcome diffidence  among those in positions of  authority and power.  There is no reason why Academia and the Scientific Establishment   should continue to   shelter  from these refreshing winds of change and transparency behind the opacity of purely subjective metrics of evaluation.

Charanjit S. Aulakh