Research Assessment

David Sweeney's new HEFCE/REF OA mandate proposal for consultation comes very close to providing the optimal OA mandate model:

(1) It separates the date on which deposit must be made (immediately upon acceptance for publication, with no differences across disciplines) from the date on which the deposit must be made OA (preferably immediately, but, at the latest, within an allowable embargo whose length will be adapted to the needs of each discipline).

(2) It specifies that the deposit must be made in the author's institutional repository (whence the metadata can be exported or harvested to institution-external discipline repositories immediately -- and the full-text once any embargo has elapsed).

(3) It makes immediate-deposit (but not immediate-OA) an eligibility precondition for submission to research evaluation (REF), thereby (very sensibly) recruiting institutions in monitoring and ensuring timely compliance with the mandate.

(4) It expresses no preference for gold OA publishing, leaving authors free to publish in whatever journal they choose.

(5) It expresses a preference for licensing certain re-use rights, but again leaves this to author choice.

I have been a strident critic of the Willetts/Finch/RCUK policy's preference for gold over green and its constraints on authors' freedom of journal choice. This new HEFCE mandate proposal would remedy all that and would make the UK's OA mandate once again compatible with green OA mandates the world over -- indeed, with (3) and (4) it provides the all-important compliance-verification mechanism that most OA mandates still lack.

I hope that once they have seriously reflected upon and understood this new mandate proposal, researchers and their institutions will see that it moots all the objections that have been raised to the Finch/RCUK mandate. And I profoundly hope that David Willetts will realize and understand that too.

I also hope that those who are impatient for immediate, embargo-free OA, CC-BY licenses and Gold OA will allow this HEFCE compromise mandate to be adopted and succeed, rather than trying to force their less urgent, less universal, and much more divisive conditions into the policy yet again.

The price of Green OA (per paper deposited) is negligibly small, compared to Gold OA. And institutional repositories are already created and paid up (for a variety of purposes) but they remain near-empty of their target OA content -- unless deposit is mandated.

Green deposit mandates have to have carrots and sticks to be effective. Funder mandates provide the carrot/stick for institutions (funding eligibility -- and enhanced impact -- if you deposit; ineligibility if you don't)

Double-paying publishers pre-emptively for gold now is fine -- if you have effectively mandated a green deposit mandate for all articles first (and you have the extra cash to double-pay publishers for subscriptions and gold).

But if you have not effectively mandated a green deposit mandate for all articles first, instead double-paying publishers pre-emptively for gold is not only a gratuitous waste of scarce research money, but a counterproductive retardant on OA growth, both in the UK and worldwide (in encouraging subscription publishers to offer hybrid gold and to increase their embargo lengths on green in order to ensure that UK authors must pick and pay for gold).

(Where gold [or a fee waiver] is offered for free to authors (& their institutions) by a journal they freely choose as suitable, authors are of course welcome to choose it -- as long as they also deposit their article in their Green OA institutional repository, just as everyone else is mandated to do.)

Global green OA grows anarchically, not journal by journal. If and when competition from green starts causing journal cancellations, journals will be forced to start cutting costs by downsizing, phasing out the obsolete print and online edition and offloading all access-provision and archiving onto the global network of green OA institutional repositories. The institutional cancellation savings will then (single-) pay for post-Green Fair Gold at an affordable, sustainable price (for peer review alone).

To instead double-pay publishers pre-emptively for gold now (in the name of "cushioning" the transition) while publishers promise to "plough back" all Gold OA double-payment into subscription savings (all publishers? all subscribers?) is simply to give publishers a license to keep charging as much as they like and never bother to do the cost-cutting and downsizing that universal mandatory green would force them to do.

If the UK double-pays for Gold pre-emptively rather than first effectively mandating Green for all UK research output, it has chosen the losing option in an unforced Prisoner's Dilemma: the UK loses and the rest of the world gains. Less an admirable moral stance or idealism or a "front-mover" advantage than an unreflective and somewhat stubborn rush for Fool's Gold.

Mark Thorley wrote: "Stevan, ...As an advocate of Open Access I would like to think that you appreciate the fact that the UK is leading the world here..."

Mark, no, the UK is no longer leading the world with its new Finch/RCUK/BIS OA policy.

It's time to heed OA advocates that have been at this far longer than you, and fix the RCUK Policy:

Peter Suber: "The UK can do better. In fact, the RCUK can do better. Its 2006 policy was better than the new policy. It only needed to be enforced."

The RCUK Policy is fixable. Indeed it can be made much better than the old RCUK policy. And the UK can once again take the worldwide lead in OA Policy:

I. Drop the 9 words that make the RCUK Policy say the opposite of what it means.

II. Adopt an effective compliance-verification mechanism for Green OA self-archiving:

(IIa) Deposit must be in the fundee's institutional repository.
      (This makes each UK institution responsible for monitoring and verifying timely compliance.)
(IIb) All articles must be deposited immediately upon acceptance for publication.
      (Publisher embargoes apply only to the date on which the deposit is made OA.)
(IIc) Repository deposit must be designated the sole mechanism for submitting publications for UK research assessment (REF).
      (Articles' deposit URL required in all requests for RCUK funding or renewal.)

It is still widely hoped that RCUK will act in a flexible, constructive way rather than a rigid, dogmatic one, in the face of the growing expression of the concerns of the research community and its OA advocates, in the UK and worldwide, about the ambiguity and the potential for perverse effects of the new RCUK OA Policy.

Stevan Harnad

BOAI-10 RECOMMENDATIONS
-- 1.1. Every institution of higher education should have a policy assuring that peer-reviewed versions of all future scholarly articles by faculty members are deposited in the institution’s designated repository...

-- Deposits should be made as early as possible, ideally at the time of acceptance, and no later than the date of formal publication.

-- University policies should respect faculty freedom to submit new work to the journals of their choice. [emphasis added]

-- University policies should encourage but not require publication in OA journals [emphasis added] ...

-- 1.3. Every research funding agency, public or private, should have a policy assuring that peer-reviewed versions of all future scholarly articles reporting funded research are deposited in a suitable repository and made OA as soon as practicable.

-- Deposits should be made as early as possible, ideally at the time of acceptance, and no later than the date of formal publication...

U Liege's Rector, Bernard Rentier, reports that over the past year deposits to the U. Liege repository (ORBi) have grown from 10 to 40 thousand publications, 25 thousand of them full-text. According to ROAR, this is the 3rd highest growth rate among the world's thousand identified institutional repositories. Viewed 650 thousand times and downloaded 61 thousand times, these 40 thousand deposits coincide with the first year in which, as a part of U Liege's Open Access Mandate, ORBi has served as U Liege's sole official means of submitting publications for performance review for academic promotion.

Universities UK recommends making all the research outputs submitted to the UK's new Research Excellence Framework (REF) Open Access (OA).

The UUK's recommendation is of course very welcome and timely.

All research funded by the RCUK research councils is already covered by the fact that all the UK councils already mandate OA. It is this policy, already adopted by the UK, that the US is now also contemplating adopting, in the form of the proposed Federal Research Public Access Act (FRPAA), as well as the discussion in President Obama's ongoing OSTP Public Access Policy Forum.

But if HEFCE were to follow the UUK's recommendation, it would help to ensure Open Access to UK research funded by the EU (for which OA is only partially mandated thus far) and other funders, as well as to unfunded research -- for which OA is mandated by a still small but growing number of universities in the UK and worldwide. (The same UUK proposal could of course be taken up by UK's universities, for once they mandate OA for all their research output, all UK research, funded and unfunded, becomes OA!)

There is an arbitrary constraint on REF submissions, however, which would greatly limit the scope of an OA requirement (as well as the scope of REF itself): Only four research outputs per researcher may be submitted, for a span covering at least four years, rather than all research output in that span.

This limitation arises because the REF retains the costly and time-consuming process of re-reviewing, by the REF peer panels, of all the already peer-reviewed research outputssubmitted. This was precisely what it had earlier been proposed to replace by metrics, if they prove sufficiently correlated with -- and hence predictive of -- the peer panel ranklings. Now it will only be partially supplemented by a few metrics.

This is a pity, and an opportunity lost, both for OA and for testing and validating a rich and diverse new battery of metrics and initializing their respective weights, discipline by discipline. Instead, UUK has endorsed a simplistic (and likewise untested and arbitrary) a-priori weighting ("60/20/20 for outputs, impact and environment").

Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1) Also in Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds. (2007)

Commentary on:
Collini, S. (2009) Impact on humanities: Researchers must take a stand now or be judged and rewarded as salesmen. Times Literary Supplement. November 13 2009.

One can agree whole-heartedly with Professor Collini that much of the spirit and the letter of the RAE and the REF and their acronymous successors are wrong-headed and wasteful -- while still holding that measures ("metrics") of scholarly/scientific impact are not without some potential redeeming value, even in the Humanities. After all, even expert peer judgment, if expressed rather than merely silently mentalized, is measurable. (Bradley's observation on the ineluctability of metaphysics applies just as aptly to metrics: "Show me someone who wishes to refute metaphysics and I'll show you a metaphysician with a rival system.")

The key is to gather as rich, diverse and comprehensive a spectrum of candidate metrics as possible, and then test and validate them jointly, discipline by discipline, against the existing criteria that each discipline already knows and trusts (such as expert peer judgment) so as to derive initial weights for those metrics that prove to be well enough correlated with the discipline's trusted existing criteria to be useable for prediction on their own.

Prediction of what? Prediction of future "success" by whatever a discipline's (or university's or funder's) criteria for success and value might be. There is room for putting a much greater weight on the kinds of writings that fellow-specialists within the discipline find useful, as Professor Collini has rightly singled out, rather than, say, success in promoting those writings to the general public. The general public may well derive more benefit indirectly, from the impact of specialised work on specialists, than from its direct impact on themselves. And of course industrial applications are an impact metric only for some disciplines, not others.

Ceterum censeo: A book-citation impact metric is long overdue, and would be an especially useful metric for the Humanities.

Harnad, S. (2001) Research access, impact and assessment. Times Higher Education Supplement 1487: p. 16.

Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives: Improving the UK Research Assessment Exercise whilst making it cheaper and easier. Ariadne 35.

Brody, T., Carr, L., Harnad, S. and Swan, A. (2007) Time to Convert to Metrics. Research Fortnight pp. 17-18.

Harnad, S. (2008) Open Access Book-Impact and "Demotic" Metrics Open Access Archivangelism October 10, 2008.

Harnad, S. (2008) Validating Research Performance Metrics Against Peer Rankings. Ethics in Science and Environmental Politics 8 (11) doi:10.3354/esep00088 Special Issue on "The Use And Misuse Of Bibliometric Indices In Evaluating Scholarly Performance"

Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1)

Open Access and Research Conference 2008

STAMFORD PLAZA HOTEL,
BRISBANE, QUEENSLAND
24-25 SEPTEMBER

"The way we create and disseminate knowledge has undergone profound change over the last ten years...

"...we have seen a worldwide move towards establishing frameworks in which we can optimise access to and reuse of research especially that which is publicly funded...

"This has been supported by the development of open access repositories, new publishing tools and models and more strategic management of copyright at the individual and institutional level.

"QUT along with many other institutions throughout the world has been a pioneer in putting in place the management practices and necessary infrastructure to promote access and innovation.

"We are proud to announce what we believe will be a truly landmark conference that will draw on experts from Australia and around the world speak on a range of topics such as evolving publishing models, repository management, e-Research, policy development, and legal and technical issues."

In addition to our line up of prominent Australian speakers, International Keynote Speakers include:

John Wilbanks, Executive Director of Science Commons

Dr Alma Swan, Founder of Key Perspectives: Consultants to the scholarly information industry

Dr Tony Hey, Corporate Vice President of the External Research Division of Microsoft Research

Professor Stevan Harnad, UQAM & School of Electronics and Computer Science, University of Southampton

Maarten Wilbers, Deputy Legal Counsel, European Organization for Nuclear Research (CERN)

Research Evaluation, Metrics and Open Access in the Humanities
Coimbra-Group Workshop
Trinity College Dublin
18-20 September 2008

-- Aimed at Arts and Humanities researchers, Deans of Research, Librarians, research group leaders and policy makers within the Coimbra-Group member universities and the Irish University sector...

-- To compare established and innovative methods and models of research evaluation and assess their appropriateness for the Arts and Humanities sector...

-- To assess the increasing impact of bibliometrical approaches and Open Access policies on the Arts and Humanities sector...

There is no need to keep waiting for governmental OA mandates.

Harnad, Stevan (2005) The OA Policy of Southampton University (ECS), UK: the "Keystroke" Strategy [Putting the Berlin Principle into Practice: the Southampton Keystroke Policy] . Delivered at Berlin 3 Open Access: Progress in Implementing the Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities, University of Southampton (UK).

University OA mandates are natural extensions of universities' existing record-keeping, asset management, and performance-assessment policies. They complement research-funder OA mandates, and are the most efficient and productive way to monitor and credit compliance and fulfillment for both. Australia's Arthur Sale has done the most work on this. Please read what he has to say:

---

Arthur Sale wrote in the American Scientist Open Access Forum:

The evidence is quite clear that advocacy does not work by itself, and never has worked anywhere. To repeat the bleeding obvious once again: depositing in repositories is avoidable work under a voluntary regime, and like all avoidable work it will be avoided by most academics, even if perceived to be in their best interests, and even if the work is minor. The work needs to be (a) required and (b) integrated into the work pattern of researchers, so it becomes the norm. This is the purpose of mandates - to make it clear to researchers that they are expected to do this work.

My research and published papers show that mandates do work, and they take a couple of years for the message to sink in. Enforcement need only be a light touch - reporting to heads of departments for example. [See references below.]

At the risk of boring some, may I point to a similar case in Australia. All universities are required to produce an annual return to the Australian Government of publications in the previous year in the categories of refereed journal articles, refereed conference papers, books, and book chapters. The universities make this known to their staff (a mandate), and they all fill out forms and provide photocopies of the works. The workload is considerably more than depositing a paper in a repository. The scheme has been going for many years and is regarded as part of the academic routine. The data is used by Government to determine part of the university block grant. The result is near 100% compliance.

What I am doing in Australia is pressing for this already existing mandate to be extended to the repositories. If the researcher deposits in the repository, and the annual return is automatically derived from the repository, then (a) the researcher wins because it takes him/her less time, (b) it takes the administrators less time as the process is automated and only needs to be audited, and (c) the repository delivers its usual benefits for those with eyes to see. All we need is for the research office to promulgate such a policy in each university. It is in their own interests as well as the university's.

Arthur Sale
University of Tasmania

Swan, A. and Brown, S. (2005) Open access self-archiving: An author study. JISC Technical Report, Key Perspectives Inc. http://eprints.ecs.soton.ac.uk/10999/

Sale, Arthur (2006) Researchers and institutional repositories, in Jacobs, Neil, Eds. Open Access: Key Strategic, Technical and Economic Aspects, chapter 9, pages 87-100. Chandos Publishing (Oxford) Limited. http://eprints.utas.edu.au/257/

Sale, A. The Impact of Mandatory Policies on ETD Acquisition. D-Lib Magazine April 2006, 12(4). http://dx.doi.org/10.1045/april2006-sale

Sale, A. Comparison of content policies for institutional repositories in Australia. First Monday, 11(4), April 2006. http://firstmonday.org/issues/issue11_4/sale/index.html

Sale, A. The acquisition of open access research articles. First Monday, 11(9), October 2006. http://www.firstmonday.org/issues/issue11_10/sale/index.html

Sale, A. (2007) The Patchwork Mandate D-Lib Magazine 13 1/2 January/February http://www.dlib.org/dlib/january07/sale/01sale.html

Posted by Arthur Sale in the American Scientist Open Access Forum:

Yesterday, Australia held a Federal Election. The Australian Labor Party (the previous opposition) have clearly won, with Kevin Rudd becoming the Prime-Minister-elect.

What has this to do with the American Scientist Open Access Forum? Well the policy of the ALP is that the plans for the Research Quality Framework (the RQF - our research assessment exercise) will be immediately scrapped, and it will be replaced by a cheaper and metrics-based assessment, presumably a year or two later.

At first sight this is a setback for open access in Australia, because institutional repositories are not essential for a metrics-based research assessment. They just help improve the metrics. However, the situation may be turned to advantage, and there are several major pluses.

(1) Previous RQF grants should have ensured that every university in Australia now has a repository. Just mostly empty, or mostly dark, or both.
(2) The advisers in the Department of Education, Science & Technology (DEST) haven’t changed. The Accessibility Framework (ie open access) is still in place as a goal.
(3) A new metric-based evaluation could and should be steered to be a multi-metric based one. The ALP has already stated that it will be discipline-dependent.
(4) If the Rudd government is serious about efficiency in higher education, they could simply instruct DEST to require universities to put all their currently reported publications in a repository (ID/OA policy), from which the annual reports would be automatically derived. In addition all the desired publication metrics would also be derived, at any time. The Accessibility Framework would be achieved.

It should now be crystal clear to every university in Australia that citations and other measures will be key in the future. It should be equally clear that they should do everything possible to increase their performance on these measures. Any university that fails to immediately implement an ID/OA mandate (Immediate Deposit, Open Access when possible) in its institutional repository is simply deciding to opt out of research competition, or mistakenly thinks that it knows better. Although I suppose there is still the weak excuse that it is all too hard to understand or think about.

Here is the edited text of a press release by the shadow minister before the election. The boldface over some paragraphs is mine.

Arthur Sale
Professor of Computer Science
University of Tasmania

[BEGINS]
Senator Kim Carr
Labor Senator for Victoria
Shadow Minister for Industry, Innovation, Science and Research

Thursday, 15 November 2007 (58/07)
Building a strong future for Australian research

Federal Labor’s key research initiatives, announced during yesterday’s Campaign Launch, highlight our commitment to a research revolution.

[snip]

A Rudd Labor Government will be committed to rebuilding the national innovation system and, over time, doubling the amount invested in R&D in Australia.

· Labor will bring responsibility for innovation, industry, science and research into a single Commonwealth Department.

· Labor will develop a set of national innovation priorities to sit over the national research priorities. Together, these will provide a framework for a national innovation system, ensuring that the objectives of research programs and other innovation initiatives are complementary.

· Labor will abolish the Howard Government’s flawed Research Quality Framework, and replace it with a new, streamlined, transparent, internationally verifiable system of research quality assessment, based on quality measures appropriate to each discipline. These measures will be developed in close consultation with the research community. Labor will also address the inadequacies in current and proposed models of research citation. Labor’s model will recognise the contribution of Australian researchers to Australia and the world.

[snip]

· Labor recognises the importance of basic research in the creation of new knowledge, and also the value and breadth of Australian research effort across the humanities, creative arts and social sciences as well as scientific and technological disciplines.

The Howard Government has allocated $87 million for the implementation of the RQF. Labor will seek to redirect the residual funds to encourage genuine industry collaboration in research.

[snip]

---

SUMMARY: Three things need to be remedied in the UK's proposed HEFCE/RAE Research Evaluation Framework:
   (1) Ensure as broad, rich, diverse and forward-looking a battery of candidate metrics as possible -- especially online metrics -- in all disciplines.
   (2) Make sure to cross-validate them against the panel rankings in the last parallel panel/metric RAE in 2008, discipline by discipline. The initialized weights can then be fine-tuned and optimized by peer panels in ensuing years.
   (3) Stress that it is important -- indeed imperative -- that all University Institutional Repositories (IRs) now get serious about systematically archiving all their research output assets (especially publications) so they can be counted and assessed (as well as accessed!), along with their IR metrics (downloads, links, growth/decay rates, harvested citation counts, etc.).
   If these three things are systematically done -- (1) comprehensive metrics, (2) cross-validation and calibration of weightings, and (3) a systematic distributed IR database from which to harvest them -- continuous scientometric assessment of research will be well on its way worldwide, making research progress and impact more measurable and creditable, while at the same time accelerating and enhancing it.

---

Once one sees the whole report, it turns out that the HEFCE/RAE Research Evaluation Framework is far better, far more flexible, and far more comprehensive than is reflected in either the press release or the Executive Summary.

It appears that there is indeed the intention to use many more metrics than the three named in the executive summary (citations, funding, students), that the metrics will be weighted field by field, and that there is considerable open-mindedness about further metrics and about corrections and fine-tuning with time. Even for the humanities and social sciences, where "light touch" panel review will be retained for the time being, metrics too will be tried and tested.

This is all very good, and an excellent example for other nations, such as Australia (also considering national research assessment with its Research Quality Framework), the US (not very advanced yet, but no doubt listening) and the rest of Europe (also listening, and planning measures of its own, such as EurOpenScholar).

There is still one prominent omission, however, and it is a crucial one:

The UK is conducting one last parallel metrics/panel RAE in 2008. That is the last and best chance to test and validate the candidate metrics -- as rich and diverse a battery of them as possible -- against the panel rankings. In all other fields of metrics -- biometrics, psychometrics, even weather forecasting metrics – before deployment the metric predictors first need to be tested and shown to be valid, which means showing that they do indeed predict what they were intended to predict. That means they must correlate with a "criterion" metric that has already been validated, or that has "face-validity" of some kind.

The RAE has been using the panel rankings for two decades now (at a great cost in wasted time and effort to the entire UK research community -- time and effort that could instead have been used to conduct the research that the RAE was evaluating: this is what the metric RAE is primarily intended to remedy).

But if the panel rankings have been unquestioningly relied upon for 2 decades already, then they are a natural criterion against which the new battery of metrics can be validated, initializing the weights of each metric within a joint battery, as a function of what percentage of the variation in the panel rankings each metric can predict.

This is called "multiple regression" analysis: N "predictors" are jointly correlated with one (or more) "criterion" (in this case the panel rankings, but other validated or face-valid criteria could also be added, if there were any). The result is a set of "beta" weights on each of the metrics, reflecting their individual predictive power, in predicting the criterion (panel rankings). The weights will of course differ from discipline by discipline.

Now these beta weights can be taken as an initialization of the metric battery. With time, "super-light" panel oversight can be used to fine-tune and optimize those weightings (and new metrics can always be added too), to correct errors and anomalies and make them reflect the values of each discipline.

(The weights can also be systematically varied to use the metrics to re-rank in terms of different blends of criteria that might be relevant for different decisions: RAE top-sliced funding is one sort of decision, but one might sometimes want to rank in terms of contributions to education, to industry, to internationality, to interdisciplinarity. Metrics can be calibrated continuously and can generate different "views" depending on what is being evaluated. But, unlike the much abused "university league table," which ranks on one metric at a time (and often a subjective opinion-based rather than an objectiveone), the RAE metrics could generate different views simply by changing the weights on some selected metrics, while retaining the other metrics as the baseline context and frame of reference.)

To accomplish all that, however, the metric battery needs to be rich and diverse, and the weight of each metric in the battery has to be initialised in a joint multiple regression on the panel rankings. It is very much to be hoped that HEFCE will commission this all-important validation exercise on the invaluable and unprecedented database they will have with the unique, one-time parallel panel/ranking RAE in 2008.

That is the main point. There are also some less central points:

The report says -- a priori -- that REF will not consider journal impact factors (average citations per journal), nor author impact (average citations per author): only average citations per paper, per department. This is a mistake. In a metric battery, these other metrics can be included, to test whether they make any independent contribution to the predictivity of the battery. The same applies to author publication counts, number of publishing years, number of co-authors -- even to impact before the evaluation period. (Possibly included vs. non-included staff research output could be treated in a similar way, with number and proportion of staff included also being metrics.)

The large battery of jointly validated and weighted metrics will make it possible to correct the potential bias from relying too heavily on prior funding, even if it is highly correlated with the panel rankings, in order to avoid a self-fulfilling prophecy which would simply collapse the Dual RAE/RCUK funding system into just a multiplier on prior RCUK funding.

Self-citations should not be simply excluded: they should be included independently in the metric battery, for validation. So should measures of the size of the citation circle (endogamy) and degree of interdisciplinarity.

Nor should the metric battery omit the newest and some of the most important metrics of all, the online, web-based ones: downloads of papers, links, growth rates, decay rates, hub/authority scores. All of these will be provided by the UK's growing network of UK Institutional Repositories. These will be the record-keepers -- for both the papers and their usage metrics -- and the access-providers, thereby maximizing their usage metrics.

REF should put much, much more emphasis on ensuring that the UK network of Institutional Repositories systematically and comprehensively records its research output and its metric performance indicators.

But overall, thumbs up for a promising initiative that is likely to serve as a useful model for the rest of the research world in the online era.

References

Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives: Improving the UK Research Assessment Exercise whilst making it cheaper and easier. Ariadne 35.

Brody, T., Kampa, S., Harnad, S., Carr, L. and Hitchcock, S. (2003) Digitometric Services for Open Archives Environments. Proceedings of European Conference on Digital Libraries 2003, pp. 207-220, Trondheim, Norway.

Harnad, S. (2006) Online, Continuous, Metrics-Based Research Assessment. Technical Report, ECS, University of Southampton.

Harnad, S. (2007) Open Access Scientometrics and the UK Research Assessment Exercise. Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds.

Brody, T., Carr, L., Harnad, S. and Swan, A. (2007) Time to Convert to Metrics. Research Fortnight pp. 17-18.

Brody, T., Carr, L., Gingras, Y., Hajjem, C., Harnad, S. and Swan, A. (2007) Incentivizing the Open Access Research Web: Publication-Archiving, Data-Archiving and Scientometrics. CTWatch Quarterly 3(3).

See also: Prior Open Access Archivangelism postings on RAE and metrics

---

The following had been based on the earlier press alone, before seeing the full report. The press release had said:

"For the science-based disciplines, a new bibliometric indicator of research quality is proposed, based on the extent to which research papers are cited by other publications. This new indicator will be combined with research income and research student data, to drive the allocation of HEFCE research funding in these disciplines"

"For the arts, humanities and social sciences (where quantitative approaches are less developed) we will develop a light touch form of peer review, though we are not consulting on this aspect at this stage."

Comments:

(1) Citations, prior funding and student counts are fine, as candidate metrics. (Citations, suitably processed and contextually weighted, are a strong candidate, but it is important not to give prior funding too high a weight, or else it collapses the UK's Dual RAE/RCUK funding system into just a multiplier effect on prior RCUK funding.

(2) But where are the rest of the candidate metrics? article counts, book counts, years publishing, co-author counts, co-citations, download counts, link counts, download/citation growth rates, decay rates, hub/authority index, endogamy index... There is such a rich and diverse set of candidate metrics. It is completely arbitrary, and unnecessary, to restrict consideration to just an a-priori three.

(3) And how and against what are the metrics to be validated? The obvious first choice is the RAE panel rankings themselves: RAE has been relying on them unquestioningly for two decades now, and the 2008 RAE will be a (last) parallel panel/metric exercise.

(4) The weights on a rich, diverse battery of metrics should be jointly validated (using multiple regression analysis) against the panel rankings to initialize the weight of each metric, and then the weights should be calibrated and adjusted field by field, with the help of peer panels.

(5) It is not at all obvious that the humanities and social sciences do not have valid, predictive metrics too: They too write and read and use and cite articles and books, receive research funding, train postgraduate students, etc.

The metric RAE should be open-minded, open-ended forward-looking, in the digital online era, putting as many candidates as possible into the metric equation, testing the outcome against panel rankings, and then calibrating them, continuously, field by field, so as to optimise them. This is not the time to commit, a priori, to just 3 metrics, not yet validated and weighted, in science fields, nor to stick to panel review in other fields, again because of a priori assumptions about its metrics.

Let 1000 metric flowers bloom, initialize their weights against the parallel panel rankings (and any other validated or face-valid criteria) and then set to work optimising those initial weights, assessing continuously, field by field, across time. The metrics should be stored and harvestable from each institution's Open Access Institutional Repository, along with the research publications and data on which they are based.

---

Stevan Harnad
American Scientist Open Access Forum