This article was actually written in March 2011. I just got around to "sounding off" about it now !
There is increasing recognition by governments world over that
the vitality and excellence of
a nation's academic and
research system plays a central and determining role in all aspects of National
well being. It particularly
affects the ability of nations
to compete effectively in the globalized economic arena of an ever
more knowledge driven world economy.
Increasing resources are being allocated to fund higher education and research and
there is a concomitant increase
in anxiety - both among the governors
and the governed - regarding the emplacement of
mechanisms of evaluation that will ensure that the resources invested
are not feed for white elephants or polish for ivory towers but rather select real talent, nurture real excellence , reward real hard work and inspire real striving : which will lead to original contributions to the growth of basic knowledge, the development of technology and the promotion of human welfare and culture in society at large. In this context the increasing use of Citation
based measures of individual and institutional performance in all developed
scientific systems challenges the Indian academic system’s continued reliance on largely subjective or
formal indices of academic and
scientific worth coupled with an often decried mediocrity in motivation, achievement
and innovation. This challenge is
particularly relevant at this conjuncture when the system needs to be
reformed to evade the stultifying effects of entrenched networks of patronage and mutual reward of academic
and scientific elites . The recent vogue of publication of Global
University rankings (by a multitude of agencies ranging from the
Times of London to the University of Western Australia or Shanghai ) which use Citation performance
of University faculty and scholars as an important component of the ranking method has also brought the issue to
the front page of public perception.
Starting from the chronic
underdevelopment of a colonial society, India
has made much progress in providing cohorts of scientific and technical staff who can contribute well when
transplanted to the scientific ecosystems of the developed world. However the achievement of scientific
excellence within the country, specially as regards advances in fundamental science – as opposed
to localization of established scientific
techniques and procedures - has been much more modest. A considerable component of the obstacles to emergence from scientific underdevelopment,
backwardness, and mediocrity
may well be the fact that hitherto the evaluation of scientific merit of individuals and
groups has largely been left to the
subjective evaluation of the very
elites whose mediocrity is lamented . It
has long been the lament of émigré
achievers that it was the certainty of being frustrated by brazenly
unfair and opaque under valuation of
their talents that drove them to quit the country. By
the very nature of science, competence to evaluate scientific work is generally restricted
to co-workers in the subfield ( for example Particle
Physics is a sub field of Physics , which is itself divided into practically autonomous subdivisions, each with it’s own standard working `lore’ , distinct sociology and accepted
norms of performance and networking.
There are divisions such as Experimental High Energy Physics, Phenomenology , Lattice Gauge Theory , and
String Theory, and it is a rare researcher in one area who is competent to
judge work in another area . Since the
density of workers in any sub-field is
rather low (typically of the order
of 1-3
individuals for any division actually present in an academic Department) , the total
pool of competent evaluators in the country as a whole is
typically less than a 100 individuals.
Naturally this very limited pool of
individuals are linked together by various professional ties and transactions as well as mutual interest in fostering
professional development including placement in
jobs, placement of students, evaluation of theses, promotions,
recognition for awards and so on .
The restriction of evaluation
to the subjective opinion of that
very set of national senior scientists and academics, whose stagnation, imitative repetitiveness and mediocrity of achievement the
Indian scientific system needs to bootstrap itself away from,
functions as a Catch-22 that retards progress.
It is an axiom of scientific administration that mediocrity perpetuates itself
by recognizing only the mediocre while
notionally genuflecting to distant
and irrelevant eminences
(the ``scientists in America have shown
syndrome’’ !).
There is thus a need to by pass this obstacle by introducing objective
factors of evaluation that cannot be
easily faked by a consensus of the mediocre, already in positions of control or eminence, whose
deficient evaluation is difficult to
successfully challenge without recourse to indices that transcend or bypass the local stagnation and
underperformance. If introduced , such mechanisms
will bias the entire system positively toward the recognition
of talent and performance and lead to
the evolution of a pool of actual (as opposed to notional) talents.
In short there is an urgent and inescapable need for objective and quantifiable inputs to the
performance evaluation process that can
escape the dangers of nepotism and
patronage influenced evaluation.
The
need for indices of achievement that are
harder to fake by collusion amongst local/national evaluators can now be partially met by taking recourse to statistical measures of scientific
achievement such as citations of research workers in research
publications by co-workers in the same field of research and various derived measures based on such
citations. Of course the mere use of
such indicators cannot replace, but only supplement, the traditional indicators of talent and
performance . Application of mind –in good faith !- to the entire available range of information
about performance is - and will
continue to be- necessary to succeed in attributing merit and rewards
in a way that will in the long run lead to the development of excellence by global standards
. Thus one should
always keep in mind the caveats that: (i) Evaluation
of performance and potential can
never become purely objective.(ii) Nor
can it escape the need for a the presumption of goodwill and faith on the part of the evaluators. (iii)
Nor can the same yardstick and
benchmarks be applied uncritically
across different subfields and
specializations. If these reservations –
and additional modulations
required in the light of subsequent experience when such metrics are
used in practice - are kept in mind then
consistent input of citation information can usefully supplement and improve evaluation .
For reasons very similar to those
adduced above in the National context , and in spite of well functioning and innovative scientific
establishments , there is also Internationally a drive towards greater reliance on innovative
statistical measures designed
to identify and rank academics and researchers according to the depth
and range of their impact on society and
peers. The increasing ability to
amass and sort vast amounts of information using the Internet has led to an qualitative change in actual and perceived role of bibliographic metrics for the evaluation
of academic performance. Of course
citation databases allow one to rapidly trace the evolution of a
scientific idea. In addition, researchers in most branches of science now
routinely use Citation Counts, H-index, Author Rank(based on the Google web page ranking algorithm !), Impact Factors and so
on to rapidly evaluate
the scientific output of candidates and peers. We shall briefly review
some of these terms below and comment on their significance.
The citation trail that an individual 's scientific publications generate is accepted as a means of ascertaining the
importance and relevance of the work done.
While scientists as individuals
may be prey to all the human frailties, nevertheless as a group
they are committed to an “inter-subjective” conception of truth :
the quest for which motivates and drives their work. The “truth'' in any subfield'' is arrived at
by a dialectical process of discussion
and criticism in the format of the scientific journals which generally
require that
the prior history of study and research be adequately summarized and
referred to as a precondition for publication. Unjustified claims of scientific seminality
or preeminence are always vigorously contested not only by those directly affected
but also by other knowledgeable
individuals in the sub-field. Thus there
is an an internal policing mechanism within each scientific sub-field to
prevent pretensions and plagiarism and thus award citation levels justified by
actual merit. Moreover no geographical area is now so dominant as to get away with recognizing only its own contributions. Any sensible and genuine scientific worker
knows he has little possibility to obtain recognition for innovation or study unless he
clearly identifies and distinguishes the work done prior to him . Thus
even if they are reluctant to cite the
works of others, specially competitors
(!) , there is really no alternative to
doing so if they have scientific priority and precedence. In this way tracking the citation counts and
connections of an individual or group is generally an effective way of tracking their actual impact
on science.
The inter-subjectivity of the scientific process has long been consciously appreciated
by scientists. However it is only over the last half century that rigorous bibliographic research has established and conceptualized how science progresses through (and therefore can be tracked by tracing) the linked percolation process connecting waves of scientific papers that inspire succeeding papers that
cite the papers that inspired them.
Such research has clearly established the “ inherent topological and graphical nature of
the worldwide citation network” as a crucial property of scientific discourse : as proposed by R. Garner in 1965.
“The use of citation counts to
rank journals was a technique used in the early part of the nineteenth century
but the systematic ongoing measurement of these counts for scientific journals
was initiated by Eugene Garfield at the Institute for Scientific Information
who also pioneered the use of these counts to rank authors and papers. In a
landmark paper of 1965 he and Irving Sher showed the correlation between
citation frequency and eminence in demonstrating that Nobel Prize winners
published five times the average number of papers while their work was cited 30
to 50 times the average.”[Wikipedia]
Since then multiple studies have re-confirmed
the strong correlation between citation frequencies and scientific originality,
relevance and influence at all levels of the achievement pyramid. Before the advent of the digital revolution
and the internet, tracking bibliographic
information and making it available for
reference by scientists or scholars was
a complex and almost prohibitive enterprise. However the Information Revolution has brought with it easy access to
large bibliographic data bases that are now essential to almost every scientist
in the internet era. Some of the well known data bases are the general purpose Science Citation Index from Thomson Reuters available online through
the Web of Science database and the SCOPUS data base available from Elsevier as a part of the ScienceDirect database .
Various specialized data bases such as the Neuroscience Citation Index from Thomson Reuters, and the path breaking and user charge free Stanford Public Information Retrieval System
for High Energy Physics (HEP-SPIRES)
are also commonly now used via the
internet thousands of times daily.
I illustrate my comments by reference to HEP-SPIRES since it is available freely on
the internet (http://www.slac.stanford.edu/spires/ and at many other ``mirror’’ sites maintained
at various high energy physics research
centers across the globe ), is complete for it’s subfield (over a million
records updated daily ) and simple to use and understand. Similar databases are now available for
Astrophysics (NASA), Chaos (Maryland), Computing(Computing reviews) , Mathematics(MathScinet, Zentralblatt
MATH), Nuclear Physics (Nuclear Science
References W3 Retrieval System ) and
so on. Commercially available data
bases such as the Web of Science and Science Direct offer many more `bells and whistles’’ on the
basic functionality of SPIRES . A successor system (INSPIRE ) has begun to
offer many of these more complex
bibliographical metrics.
As an
example we give in Table 1 the results of a search of the
HEP-SPIRES/INSPIRE database for
renowned Indian Theoretical High
Physicist Asoke Sen’s bibliographic data. The search instantly yields a summary page that shows just how
high the performance bar can be objectively set even for scientists working within the
country. The summary reveals an
extraordinary level of scientific productivity, excellence and information. Over 212 scientific papers of which 202 are
papers published in recognized (by HEP-SPIRES criteria) scientific media and
have been cited over 18000 times by various
scientific articles. With an average number of citations per
paper of 86 and an extraordinary number of
over 100 ``TOPCITE’’ papers ( two with more than 500 but less than
700 citations , 19 with 250-499
citations , 30 with 100-249
citations and so on) the impressive
citation counts are supplemented by his extraordinarily high
H(for Hirsch)-index of 75. The H-index is defined as the maximal
number H
of papers with H or more citations each
for the author in question. Thus
this author has at 75 , but not
76, papers with 75 or more citations each. It is known that the total number of
citations $N$ scales with the square of
the h-index (with a coefficient of
3.5-4) .
|
Citation summary results
|
All papers
|
Published only
|
|
Total number of citable
papers analyzed:
|
212
|
202
|
|
Total number of citations:
|
18,501
|
17,534
|
|
Average citations per
paper:
|
87.3
|
86.8
|
|
Breakdown of papers by citations:
|
|
|
|
Renowned papers (500+)
|
2
|
2
|
|
Famous papers (250-499)
|
19
|
18
|
|
Very well-known papers (100-249)
|
32
|
31
|
|
Well-known papers (50-99)
|
48
|
45
|
|
Known papers (10-49)
|
86
|
82
|
|
Less known papers (1-9)
|
22
|
21
|
|
Unknown papers (0)
|
3
|
3
|
|
|
|
|
Additional Citation Metrics
|
|
|
|
h-index
|
75
|
73
|
Table 1 : Results of a search for citation data of
renowned Indian String Theorist Asoke Sen on March 13, 2011.
It is interesting to contrast
this record of a largely Mathematically
oriented physicist with that of the three giants who defined the Standard Model
of Particle physics : Steven Weinberg (184 papers, 49012 citations, H-index 94,
114 topcites, 1 paper cited more than 7000 times one more than 2000 times and 7
more than 1000 times), Abdus Salam (232
papers, 15679 citations, H-index 53, 58 topcites, 1 paper cited more than 3000
times and one more
than 1000 times) and Sheldon Glashow
(150 papers, 26921 citations, H-index 59, 65 topcites, 2 papers cited
more than 4000 times and one each more
than 3000, 2000 and 1000 times respectively).
At least to those knowledgeable
about these subjects the citation
levels and details contain a wealth of
information on the contributions and
stature of these scientists and the sociology of their fields of research.
A use of the freely available SPIRES database to study the
citation levels of various other more or less eminent Indian High Energy Physicists is also most instructive. One finds instances
of eminences, members of various National Academies and Directors
of National institutes who have
accumulated only a few hundred citations in a long career spanning forty years yet
habitually sit in exacting judgment on scientists 30 years their junior who
already have better recognized work as
far as citation levels go! Like the dynamic
effects of the recently introduced Right
to Information laws on entrenched bureaucracies and patronage
networks , citation information
has the potential to upset many
comfortable academic apple carts of the powerful but often comatose guardians of entrenched mediocrity in our
academic and research system .
The HEP-SPIRES data base allows
various methods for further refining the
metrics presented to remove doubts whether the citation record presents a true picture of the esteem in
which the work done by the author is
actually held by his scientific peers.
For instance one may
subtract self citations to
check whether the citation picture changes drastically thereby. Another
check that can prove very revealing is the number of cites divided
by the number of authors. In fields such
as Experimental High Energy Physics where
teams of thousands of individuals labour on an experiment and publish
collectively it often happens that the
papers of the collaboration are cited thousands of times. Similarly when a
working group of tens or hundreds of theorists publishes a ``white paper’’
summarizing a consensus in an evolving field it often receives a large number
of citations. In such situations a re-normalization is called for in order. For example
a division by the number of authors
can however yield a figure of only a few citations per author ! Clearly
the true picture regarding the scientific contribution of the individual
scientists in the huge collaborations
must be sought by other means than a simple citation count. In fact there exist methods in such fields for
obtaining a true picture of an individuals talent and contribution by studying
the Collaboration Notes contributed by
small sub-groups and their citation by the final published paper, opinions of group leaders etc. This
example underlines the dangers of uninformed
use of citation numbers, specially
when making comparisons between different subjects. Thus the use of citation data requires an
informed formulation of appropriate
norms for citation performance in each field separately. The reader who repeats the above
exercise (performed using the
INSPIRE database at http://inspirebeta.net/search) will also
soon realize that the automated
databases are bedeviled by familiar digital bugbears : the need to discriminate
between individuals carrying closely similar names or initials. There is thus also a need to provide unique identity labels to authors which has
not been addressed satisfactorily so far.
Another source of confusion is
the use of Journal Impact Factors to judge the excellence of individuals .
Some administrators have begun to
reward and recognize faculty on the basis of Cumulative (Journal) Impact Factors. To appreciate the absurdity and negative impact of such a policy one
should first be clear about what Impact
Factors actually refer to. Unfortunately many Indian academics have only a hazy idea of the meaning of this terms, or perhaps actually support the introduction
of this policy for even less
admirable reasons. The Wikipedia defines
the Impact Factor(IF) as follows :
"In a given year, the impact factor of a journal is the
average number of citations to those papers that were published during the two
preceding year. For example,
the 2003 impact factor of a journal would be calculated as follows: A =
the number of times articles published in 2001 and 2002 were cited by indexed
journals during 2003. B = the total number of "citable items"
published in 2001 and 2002. ("Citable items" are usually articles,
reviews, proceedings, or notes; not editorials or Letters-to-the-Editor.) .2003
impact factor = A/B”
It also
succinctly describes the reasons for the absurdity of using IF to judge
individuals and how this fact is also widely appreciated by administrators in
advanced countries :
"The
impact factor is often misused to evaluate the importance of an individual
publication or evaluate an individual researcher. This does not work well since
a small number of publications are cited much more than the majority - for
example, about 90% of Nature's 2004 impact factor was based on only a quarter
of its publications, and thus the importance of any one publication will be
different from, and in most cases less than, the overall number. The impact
factor, however, averages over all articles and thus underestimates the
citations of the most cited articles while exaggerating the number of citations
of the majority of articles. Consequently, the Higher Education Funding Council
for England was urged by the House of Commons Science and Technology Select
Committee to remind Research Assessment
Exercise panels that they are obliged to assess the quality of the content of
individual articles, not the reputation of the journal in which they are
published."
Note that a policy that spuriously uses an index
appropriate for journals to judge individuals is not only unjust but also is clearly anti-National inasmuch it seeks to introduce a
bias that will tend to prevent Indian
researchers from publishing in national journals such as Pramana since these
do not have anywhere near the same IF as many other
journals. Even a person who performs work
that is well cited (even though
published in a low IF journal) will not
be recognized under such policies. Indeed over time all sub-dominant If journals
would be driven towards a fixed point of low or zero Impact Factor ! The logic behind such thinking seems to
be that of an outsider who recounts the instances when he manages to
penetrate an exclusive club as a proof
of his social merit : even if the club itself may rue the lowering of it’s
tone(i.e of it’s IF !) by his entry (publication in that journal) !
Like all reasoning based on snobbery,
whether direct or inverted, it cannot lead
to healthy development or behaviour and
specially will not encourage individuals to
mount the challenges to orthodoxy and dominance that are the heart of good science. Rather it is a genuflection to
the established scientific dominance order which countries like India need to challenge if
they are ever to come into their own.
|
In spite of these facts being well
known and critiques being easily
accessible, administrators in the
Indian context continue to use IF and CIF as measures of individual
performance. It also bears mention
that the detailed evaluation criteria
published by the UGC as a part of the revision of Pay scales under the 6th
Pay Commission make no mention of Citation levels as a basis for judgment of research performance . Little interest is evinced by teachers and
researchers in Universities and National Institutions towards a reform of the evaluation system to include
citation data to provide a more accurate and objective picture of individual performance. Nor has the manifest perversity of using IF
and CIF as opposed to more appropriate measures like cumulative citation counts and H-index, average citations per paper etc
elicited much protest . Indeed one
gathers the impression that many
Indian academics are motivated to cast
doubts on the citation based
evaluation methodologies using arguments based on real or imagined
anomalies that may occur and thus argue against even the introduction of the use of citation data.
It is amusing, however, that often the very same critics of citation
based evaluation are capable of waxing
eloquent upon the influence of their own papers when they gain even a modicum
of citations. We have already
mentioned that various caveats are in order
when using citation data but it
is our firm belief that the use of
citation data is essential and inevitable at least for those sectors of
the system that have performance
levels that are not so low as to escape evaluation on the basis of citations
completely ! India can ill afford an
ostrich like attitude towards such a
positive and enabling technological
development which has the potential to debunk and dis-empower entrenched but
dysfunctional elites and turbo-charge
our evolution as a Global scientific power. Introduction and publication of citation
data relevant to individuals selected for any scientific or academic
position(and their unsuccessful competitors !) would go a long way in reforming the
academic evaluation system . Such an information and
driven reform would be consonant with the spirit of the
age as already seen in the radical effects of the Right to Information and
Asset Disclosure laws which have
already opened the working of the Bureaucracy , Legislature and Judiciary to
public oversight and reform and induced a welcome diffidence among those in positions of authority and power. There is no reason why Academia and the Scientific
Establishment should continue to shelter from these refreshing winds of change and
transparency behind the opacity of purely subjective metrics of evaluation.
|
Charanjit
S. Aulakh
