Wednesday, December 17. 2014
Steven Hill of HEFCE has posted “an overview of the work HEFCE are currently commissioning which they are hoping will build a robust evidence base for research assessment” in LSE Impact Blog 12(17) 2014 entitled Time for REFlection: HEFCE look ahead to provide rounded evaluation of the REF
Let me add a suggestion, updated for REF2014, that I have made before (unheeded):
Scientometric predictors of research performance need to be validated by showing that they have a high correlation with the external criterion they are trying to predict. The UK Research Excellence Framework (REF) -- together with the growing movement toward making the full-texts of research articles freely available on the web -- offer a unique opportunity to test and validate a wealth of old and new scientometric predictors, through multiple regression analysis: Publications, journal impact factors, citations, co-citations, citation chronometrics (age, growth, latency to peak, decay rate), hub/authority scores, h-index, prior funding, student counts, co-authorship scores, endogamy/exogamy, textual proximity, download/co-downloads and their chronometrics, tweets, tags, etc.) can all be tested and validated jointly, discipline by discipline, against their REF panel rankings in REF2014. The weights of each predictor can be calibrated to maximize the joint correlation with the rankings. Open Access Scientometrics will provide powerful new means of navigating, evaluating, predicting and analyzing the growing Open Access database, as well as powerful incentives for making it grow faster.
Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1)
(Also in Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds. 2007)
The Only Substitute for Metrics is Better Metrics (2014)
On Metrics and Metaphysics (2008)
REF2014 gives the 2014 institutional and departmental rankings based on the 4 outputs submitted.
That is then the criterion against which the many other metrics I list below can be jointly validated, through multiple regression, to initialize their weights for REF2020, as well as for other assessments. In fact, open access metrics can be — and will be — continuously assessed, as open access grows. And as the initialized weights of the metric equation (per discipline) are optimized for predictive power, the metric equation can replace the peer rankings (except for periodic cross-checks and updates) -- or at least supplement it.
Single metrics can be abused, but not only can abuses be named and shamed when detected, but it becomes harder to abuse metrics when they are part of a multiple, inter-correlated vector, with disciplinary profiles of their normal interactions: someone dispatching a robot to download his papers would quickly be caught out when the usual correlation between downloads and later citations fails to appear. Add more variables and it gets even harder.
In a weighted vector of multiple metrics like the sample I had listed, it’s no use to a researcher if told in advance that for REF2020 the metric equation will be the following, with the following weights for their particular discipline:
The potential list could be much longer, and the weights can be positive or negative, and varying by discipline.
"The man who is ready to prove that metric knowledge is wholly impossible… is a brother metrician with rival metrics…”
Thursday, July 25. 2013
"The UK funding councils have narrowed the scope of their proposed open access mandate for the post-2014 research excellence framework."1. Model. The HEFCE proposal to mandate immediate (not retrospective) deposit of journal articles in the author's institutional repository in order to make them eligible for evaluation in the next Research Excellence Framework (REF) is wise and timely, and, if adopted, will serve as a model for the rest of the world. It will also complement the Green (self-archiving) component of the RCUK Open Access (OA) mandate, providing it with an all-important mechanism for monitoring and ensuring compliance.
2. Monographs. Exempting monographs for now was a good decision. The HEFCE mandate, like the RCUK mandate, applies only to peer-reviewed journal articles. These are all author giveaways, written solely for research impact, not royalty income. This is not true of all monographs. (But a simple compromise is possible: recommend -- but don't require -- monograph deposit too, but with access set as Closed Access rather than Open Access, with no limit on the length of the OA embargo. Author choice.)
3. Data. Ditto for open data: It's good judgment not to force it on researchers. Researchers must be allowed a fair period of first-expoitation rights on the data they have gathered. If it's immediately open to all, why bother to gather data at all? Just analyze the data of others immediately after they take the time and trouble to gather it. (But here too, a simple compromise would be to recommend -- but not require -- Closed Access deposit. Eventually, fair embargo length limits can be decided, on a discipline by discipline and project by project basis.)
4. Exceptions. The required compliance rate has not been reduced from 100% to 60-75% (and should not be). HEFCE is merely asking in the consultation, whether the research community prefers a reduced target percentage or case-by-case consideration of exceptions. The latter is a far better way of making the policy realistic and successful. Most of the notional reasons for non-compliance (e.g., publisher embargoes) are based on misunderstandings anyway. (Articles can be deposited immediately, even if there is a publisher OA embargo: access to the immediate-deposit can be set as Closed Access instead of OA during the embargo.) Percentage-targets would simply ensure that compliance rates were no higher than the allowable percentages.
5. Embargoes. The HEFCE mandate moots OA embargoes because it requires immediate deposit, whether or not access is immediately OA. This is the core reason the HEFCE mandate is so very important and provides an optimal mandate model for the rest of the world: Publisher OA embargoes no longer determine whether and when an article is deposited. And the institutional repositories have an eprint request Button with which individual users worldwide can request a single copy of a Closed Access article for research purposes with one click; and the author can choose to comply or not comply with one click. This tides over research needs during any allowable OA embargo with "Almost-OA."
6. Licenses. Once the allowable embargo (if any) elapses, any OA deposit can be accessed, read, searched, linked, downloaded, stored, printed off or locally data-mined by any user webwide. It will also be harvested and indexed for Boolean full text search by engines like Google. No further license is needed for any of this. Further re-use rights will come once effective Green OA mandates on the combined HEFCE/RCUK model are adopted globally by funders and institutions worldwide. Universal Green OA will also hasten the inevitable natural demise of all remaining OA embargoes.
7. Start-Date. The HEFCE consultation also inquires about when the mandate should start, and contemplates a grace period of two years, from 2014-2016. But there is really no reason why an immediate-deposit mandate for REF 2020 should not start immediately after REF 2014 for authors at UK institutions, for any article accepted after that date: Everyone begins preparing for the new REF the day after the old REF anyway.
8. Date-Stamp: Only one of the consultation questions is critical for the success of the HEFCE mandate model, and that is whether the requirement that the deposit be "immediate" refers to the date of publication or the date of acceptance for publication. It is crucially important that the date should be acceptance, not publication. Acceptance date is marked by a determinate date-stamped acceptance letter and is a natural point for deposit in the author's workflow. Authors usually don't even know when their accepted article will appear, or has appeared; the lag may be months or even years from acceptance. Nor is the date on the journal issue a marker, because issues often appear long after their calendar dates. Publication lags can be even longer than OA embargoes! Meanwhile, precious access and impact are being lost. The HEFCE immediate-deposit mandate will only succeed if it is pegged to the determinate acceptance date rather than the indeterminate publication date.
[Please post your own response to the HEFCE REF OA Policy Consultation HERE]
Executive Summary:I. The HEFCE proposal to mandate immediate repository deposit of articles as a precondition for eligibility for REF is excellent. If adopted and effectively implemented, it will serve as a model for OA mandates worldwide. It will also reinforce and complement the RCUK OA mandate, providing it with a uniform compliance monitoring and verification mechanism.
Question 1: Do you agree that the criteria for open access are appropriate (subject to clarification on whether accessibility should follow immediately on acceptance or on publication)?
Question 2a: Do you agree with the role outlined for institutional repositories, subject to further work on technical feasibility?
Question 2b: Should the criteria require outputs to be made accessible through institutional repositories at the point of acceptance or the point of publication?
Deposit should definitely be required at point of acceptance rather than at point of publication, for the following reasons:
Question 3a: Do you agree that the proposed embargo periods should apply by REF main panel?
Question 3b: Do you agree with the proposed requirements for appropriate licences?
Question 4: Do you agree that the criteria for open access should apply only to journal articles and conference proceedings for the post-2014 REF?
Question 5: Do you agree that a notice period of two years from the date of the policy announcement is appropriate to allow for the publication cycle of journal articles and conference proceedings?
Question 6: Do you agree that criteria for open access should apply only to those outputs listing a UK HEI in the output’s ‘address’ field for the post-2014 REF?
Question 7: Which approach to allowing exceptions is preferable?
I support Option a: Full compliance; exceptions considered on case by case basis, first by the HEI, and if not resolved, by the REF panel.
Thursday, February 28. 2013
David Sweeney's new HEFCE/REF OA mandate proposal for consultation comes very close to providing the optimal OA mandate model:
(1) It separates the date on which deposit must be made (immediately upon acceptance for publication, with no differences across disciplines) from the date on which the deposit must be made OA (preferably immediately, but, at the latest, within an allowable embargo whose length will be adapted to the needs of each discipline).I have been a strident critic of the Willetts/Finch/RCUK policy's preference for gold over green and its constraints on authors' freedom of journal choice. This new HEFCE mandate proposal would remedy all that and would make the UK's OA mandate once again compatible with green OA mandates the world over -- indeed, with (3) and (4) it provides the all-important compliance-verification mechanism that most OA mandates still lack.
I hope that once they have seriously reflected upon and understood this new mandate proposal, researchers and their institutions will see that it moots all the objections that have been raised to the Finch/RCUK mandate. And I profoundly hope that David Willetts will realize and understand that too.
I also hope that those who are impatient for immediate, embargo-free OA, CC-BY licenses and Gold OA will allow this HEFCE compromise mandate to be adopted and succeed, rather than trying to force their less urgent, less universal, and much more divisive conditions into the policy yet again.
The price of Green OA (per paper deposited) is negligibly small, compared to Gold OA. And institutional repositories are already created and paid up (for a variety of purposes) but they remain near-empty of their target OA content -- unless deposit is mandated.
Green deposit mandates have to have carrots and sticks to be effective. Funder mandates provide the carrot/stick for institutions (funding eligibility -- and enhanced impact -- if you deposit; ineligibility if you don't)
Double-paying publishers pre-emptively for gold now is fine -- if you have effectively mandated a green deposit mandate for all articles first (and you have the extra cash to double-pay publishers for subscriptions and gold).
But if you have not effectively mandated a green deposit mandate for all articles first, instead double-paying publishers pre-emptively for gold is not only a gratuitous waste of scarce research money, but a counterproductive retardant on OA growth, both in the UK and worldwide (in encouraging subscription publishers to offer hybrid gold and to increase their embargo lengths on green in order to ensure that UK authors must pick and pay for gold).
(Where gold [or a fee waiver] is offered for free to authors (& their institutions) by a journal they freely choose as suitable, authors are of course welcome to choose it -- as long as they also deposit their article in their Green OA institutional repository, just as everyone else is mandated to do.)
Global green OA grows anarchically, not journal by journal. If and when competition from green starts causing journal cancellations, journals will be forced to start cutting costs by downsizing, phasing out the obsolete print and online edition and offloading all access-provision and archiving onto the global network of green OA institutional repositories. The institutional cancellation savings will then (single-) pay for post-Green Fair Gold at an affordable, sustainable price (for peer review alone).
To instead double-pay publishers pre-emptively for gold now (in the name of "cushioning" the transition) while publishers promise to "plough back" all Gold OA double-payment into subscription savings (all publishers? all subscribers?) is simply to give publishers a license to keep charging as much as they like and never bother to do the cost-cutting and downsizing that universal mandatory green would force them to do.
If the UK double-pays for Gold pre-emptively rather than first effectively mandating Green for all UK research output, it has chosen the losing option in an unforced Prisoner's Dilemma: the UK loses and the rest of the world gains. Less an admirable moral stance or idealism or a "front-mover" advantage than an unreflective and somewhat stubborn rush for Fool's Gold.
Saturday, September 29. 2012
Mark Thorley wrote: "Stevan, ...As an advocate of Open Access I would like to think that you appreciate the fact that the UK is leading the world here..."Mark, no, the UK is no longer leading the world with its new Finch/RCUK/BIS OA policy.
It's time to heed OA advocates that have been at this far longer than you, and fix the RCUK Policy:
Peter Suber: "The UK can do better. In fact, the RCUK can do better. Its 2006 policy was better than the new policy. It only needed to be enforced."The RCUK Policy is fixable. Indeed it can be made much better than the old RCUK policy. And the UK can once again take the worldwide lead in OA Policy:
I. Drop the 9 words that make the RCUK Policy say the opposite of what it means.It is still widely hoped that RCUK will act in a flexible, constructive way rather than a rigid, dogmatic one, in the face of the growing expression of the concerns of the research community and its OA advocates, in the UK and worldwide, about the ambiguity and the potential for perverse effects of the new RCUK OA Policy.
Tuesday, June 1. 2010
U Liege's Rector, Bernard Rentier, reports that over the past year deposits to the U. Liege repository (ORBi) have grown from 10 to 40 thousand publications, 25 thousand of them full-text. According to ROAR, this is the 3rd highest growth rate among the world's thousand identified institutional repositories. Viewed 650 thousand times and downloaded 61 thousand times, these 40 thousand deposits coincide with the first year in which, as a part of U Liege's Open Access Mandate, ORBi has served as U Liege's sole official means of submitting publications for performance review for academic promotion.
Wednesday, January 6. 2010
Universities UK recommends making all the research outputs submitted to the UK's new Research Excellence Framework (REF) Open Access (OA).
The UUK's recommendation is of course very welcome and timely.
All research funded by the RCUK research councils is already covered by the fact that all the UK councils already mandate OA. It is this policy, already adopted by the UK, that the US is now also contemplating adopting, in the form of the proposed Federal Research Public Access Act (FRPAA), as well as the discussion in President Obama's ongoing OSTP Public Access Policy Forum.
But if HEFCE were to follow the UUK's recommendation, it would help to ensure Open Access to UK research funded by the EU (for which OA is only partially mandated thus far) and other funders, as well as to unfunded research -- for which OA is mandated by a still small but growing number of universities in the UK and worldwide. (The same UUK proposal could of course be taken up by UK's universities, for once they mandate OA for all their research output, all UK research, funded and unfunded, becomes OA!)
There is an arbitrary constraint on REF submissions, however, which would greatly limit the scope of an OA requirement (as well as the scope of REF itself): Only four research outputs per researcher may be submitted, for a span covering at least four years, rather than all research output in that span.
This limitation arises because the REF retains the costly and time-consuming process of re-reviewing, by the REF peer panels, of all the already peer-reviewed research outputssubmitted. This was precisely what it had earlier been proposed to replace by metrics, if they prove sufficiently correlated with -- and hence predictive of -- the peer panel ranklings. Now it will only be partially supplemented by a few metrics.
This is a pity, and an opportunity lost, both for OA and for testing and validating a rich and diverse new battery of metrics and initializing their respective weights, discipline by discipline. Instead, UUK has endorsed a simplistic (and likewise untested and arbitrary) a-priori weighting ("60/20/20 for outputs, impact and environment").
Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1) Also in Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds. (2007)
Saturday, November 28. 2009
Commentary on:One can agree whole-heartedly with Professor Collini that much of the spirit and the letter of the RAE and the REF and their acronymous successors are wrong-headed and wasteful -- while still holding that measures ("metrics") of scholarly/scientific impact are not without some potential redeeming value, even in the Humanities. After all, even expert peer judgment, if expressed rather than merely silently mentalized, is measurable. (Bradley's observation on the ineluctability of metaphysics applies just as aptly to metrics: "Show me someone who wishes to refute metaphysics and I'll show you a metaphysician with a rival system.")
The key is to gather as rich, diverse and comprehensive a spectrum of candidate metrics as possible, and then test and validate them jointly, discipline by discipline, against the existing criteria that each discipline already knows and trusts (such as expert peer judgment) so as to derive initial weights for those metrics that prove to be well enough correlated with the discipline's trusted existing criteria to be useable for prediction on their own.
Prediction of what? Prediction of future "success" by whatever a discipline's (or university's or funder's) criteria for success and value might be. There is room for putting a much greater weight on the kinds of writings that fellow-specialists within the discipline find useful, as Professor Collini has rightly singled out, rather than, say, success in promoting those writings to the general public. The general public may well derive more benefit indirectly, from the impact of specialised work on specialists, than from its direct impact on themselves. And of course industrial applications are an impact metric only for some disciplines, not others.
Ceterum censeo: A book-citation impact metric is long overdue, and would be an especially useful metric for the Humanities.
Harnad, S. (2001) Research access, impact and assessment. Times Higher Education Supplement 1487: p. 16.
Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives: Improving the UK Research Assessment Exercise whilst making it cheaper and easier. Ariadne 35.
Brody, T., Carr, L., Harnad, S. and Swan, A. (2007) Time to Convert to Metrics. Research Fortnight pp. 17-18.
Harnad, S. (2008) Open Access Book-Impact and "Demotic" Metrics Open Access Archivangelism October 10, 2008.
Harnad, S. (2008) Validating Research Performance Metrics Against Peer Rankings. Ethics in Science and Environmental Politics 8 (11) doi:10.3354/esep00088 Special Issue on "The Use And Misuse Of Bibliometric Indices In Evaluating Scholarly Performance"
Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1)
Tuesday, September 9. 2008
Open Access and Research Conference 2008
-- Aimed at Arts and Humanities researchers, Deans of Research, Librarians, research group leaders and policy makers within the Coimbra-Group member universities and the Irish University sector...Research Evaluation, Metrics and Open Access in the Humanities
-- To compare established and innovative methods and models of research evaluation and assess their appropriateness for the Arts and Humanities sector...
-- To assess the increasing impact of bibliometrical approaches and Open Access policies on the Arts and Humanities sector...
(Page 1 of 3, totaling 29 entries) » next page
Syndicate This Blog
Materials You Are Invited To Use To Promote OA Self-Archiving:
The American Scientist Open Access Forum has been chronicling and often directing the course of progress in providing Open Access to Universities' Peer-Reviewed Research Articles since its inception in the US in 1998 by the American Scientist, published by the Sigma Xi Society.
The Forum is largely for policy-makers at universities, research institutions and research funding agencies worldwide who are interested in institutional Open Acess Provision policy. (It is not a general discussion group for serials, pricing or publishing issues: it is specifically focussed on institutional Open Acess policy.)
You can sign on to the Forum here.