Open Access Archivangelism - Peer Review

Accessibility, Affordability and Quality: Think It Through

nospam@example.com (Stevan Harnad) — Sat, 03 Nov 2012 13:00:30 +0000

Reply to Ross Mounce:

1.The affordability problem loses all of its importance and urgency once globally mandated Green OA has its dual effect of (i) making peer-reviewed journal articles free for all (not just subscribers), thereby (ii) making it possible for institutions to cancel subscriptions if they can no longer afford -- or no longer wish -- to pay for them:

2. "If and when global Green OA should go on to make subscriptions unsustainable (because users are satisfied with just the Green OA versions) that will in turn induce journals to cut costs (print edition, online edition, access-provision, archiving), downsize to just providing the service of peer review, and convert to the Gold OA cost-recovery model. Meanwhile, the subscription cancellations will have released the funds to pay these residual service costs."

3. In other words, it is global Green OA itself that will "decouple the scholarly journal, and separate the peer-review process from the integrated set of services that traditional journals provide." (And then the natural way to charge for the service of peer review will be on a "no-fault basis," with the author's institution or funder paying for each round of refereeing, regardless of outcome [acceptance, revision/re-refereeing, or rejection], minimizing cost while protecting against inflated acceptance rates and decline in quality standards.)

4. Because of the Gaussian distribution of virtually all human qualities and quantities, research quality and quality-assessment is not just a 0/1, pass/fail matter. Research and researchers need the much more nuanced and informative hierarchy of quality levels that journals afford.

5. Please don't conflate the simplistic reliance on the journal impact factor (the journal's articles' average citation counts) -- which is not, by the way, an OA issue -- with the much more substantive and important fact that the existing journal hierarchy does represent a vertical array of research quality levels, corresponding to different standards of peer-review rigour and hence selectivity.

6. This hierarchy is already provided by the known track-records of journals, and can and will be enhanced by a growing set of new, rich and diverse metrics of journal and article quality, importance, usage and impact. What authors and users need is not just a "Gold OA plot" but a clear sense of the quality standards of journals.

7. I think it is exceedingly unrealistic and counterproductive to advise researchers to simply give up the journal with the known and established quality standards and track-record for their work in favour of another journal just for the sake of making their paper Gold OA (let alone for the sake of freeing it from the tyranny of the impact factor) -- and especially at today's still vastly-inflated Gold OA "publishing fee," and whilst the money to pay for it is still locked into institutional subscriptions that cannot be cancelled until/unless those journal articles are accessible in some other way.

8. That other way is cost-free Green OA. And it is global Green OA self-archiving of all journal articles, published in the journal with the highest quality standards the author's work can meet -- not a pre-emptive switch to new journals just because they offer Gold OA today -- that will make those journal articles accessible in the "other way" that (i) solves the accessibility problem immediately, (ii) mitigates the affordability problem immediately, and (iii) eventually induces a transition to Gold OA at a fair and affordable price.

9. Today (i.e., pre-Green-OA), Gold OA means double-pay -- whether for hybrid Gold OA, or for pure Gold OA, as long as subscriptions must still be paid too.

10. It is short-sighted in the extreme to wish authors to renounce journals of established quality and pay extra pre-emptively to new Gold OA journals for an OA that they can already provide cost-free today through Green OA self-archiving, with the additional prospect of easing the affordability problem now, as well as preparing the road for an eventual liberation from subscriptions and a leveraged transition to affordable, sustainable Gold (and Libre) OA.

Harnad, S. (2007) The Green Road to Open Access: A Leveraged Transition. In: Anna Gacs. The Culture of Periodicals from the Perspective of the Electronic Age. L'Harmattan. 99-106.

Harnad, S. (2009) The PostGutenberg Open Access Journal. In: Cope, B. & Phillips, A (Eds.) The Future of the Academic Journal. Chandos.

Harnad, S. (2010) No-Fault Peer Review Charges: The Price of Selectivity Need Not Be Access Denied or Delayed. D-Lib Magazine 16 (7/8).

Harnad, S (2012) The Optimal and Inevitable outcome for Research in the Online Age. CILIP Update September 2012

WikiWeakPoints: Notability vs. Noteworthiness, Anonymity vs. Answerability

nospam@example.com (Stevan Harnad) — Sun, 20 Nov 2011 13:55:45 +0000

Re: "Why are pornstars more notable than scientists on Wikipedia?" and "Wikipedian in Residence"

Just a small sample of the patently obvious and persistent fallacies in the notion that anonymous global cloud-writing can produce reliable information on anything that's more than skin-deep (I could go on and on and on):

(1) A neutral point of view on what is true?

(2) Expertise is no excuse?

(3) Expertise is elitism?

(4) Expertise is bias?

(5) Write on what you don't know?

(6) The longer your track-record of being a dilettante busybody, the more decision power you merit?

(7) Zipf's Law trumps the Matthew Effect?

(8) Notability, not noteworthiness, rules?

(9) Anonymous gallup polls, not personal answerability, keep people honest and on their toes?

(10) Crowd-sourcing protects against regression on the mean?

(11) Porifera is on a par with porn?

The surprise is not when Wikipedia gets things wrong, but when it gets them right.

(And the only virtue of notability is that it reduces the motivation of most wikipedia busybodies to bother with esoteric scientific and scholarly topics. Trouble is that it just takes one officious dilettante with a long wack-record to cast a contagious shadow of doubt over stuff he doesn't know, understand or care about.)

My guess is that the only reason any qualified experts even bother to have a go at writing in Wikipedia is Wikipedia's PageRank notoriety, which influences students and public opinion as their first (and often only) port of call.

The only hope is that Open Access to the primary scientific and scholarly literature will remedy that, leaving Wikipedia to rule where it really is the expert: Trivial Pursuit.

Stevan Harnad

Post-Peer-Review Open Access, Commentary and Metrics versus Post Hoc "Peer Review"

nospam@example.com (Stevan Harnad) — Wed, 07 Sep 2011 14:11:52 +0100

David Colquhoun (2011) is quite right on practically every point he makes: There is too much pressure to publish and too much emphasis on journal impact factors. Too many papers are published. Many are not worth publishing (trivial or wrong). Peer reviewers are overworked. Refereeing is not always reliable. There is a hierarchy of journal peer review quality. The lower levels of the quality hierarchy are practically unrefereed.

But the solution is not to post everything publicly first, and entrust the rest to post hoc "peer review," including anonymous peer review.

First, the situation is not new. Already a quarter century ago Stephen Lock (1985), former editor of the British Medical Journal, noted that everything was getting published, somewhere, in the hierarchy of journals.

And journals' positions in the hierarchy serve a purpose: Their names and public track-records for quality are important filters for potential users, helping them decide what to invest their limited time in reading, how much to trust it, and whether to risk trying to apply or build upon it. This is especially true in medicine, where it is not just researchers' time and careers that are at risk from invalid results.

Classical, prepublication peer review is answerable: The submitted paper is answerable to the referees. The referees are answerable to the editors. The editors and journal are answerable to the readership. In the higher quality journals, if revisions cannot be made to bring a paper up to its standards, it is rejected. Peer review is a means of raising paper quality, for authors, and of filtering paper quality, for users.

Self-appointed post hoc peer review is not answerable. No editor's or journal's public track record is at stake to ensure that qualified referees assess the papers, nor that their recommendations for revision are valid, heeded or followed.

And referee anonymity is a two-edged sword. Yes, it protects junior researchers and rivals from vindictiveness, but it also allows anyone to say anything about anything, immunized by anonymity. (Look at the unevenness in the quality of the comments on Professor Colquhoun's article here in the Guardian. This is not peer review.) Journal referees are anonymous to authors, but not to editors.

No, the solution is not that everything should be publicly posted, unrefereed, and then to hope that open commentary will somehow take care of the rest. The solution is to post all peer-reviewed, revised and accepted papers online, free for all (Open Access) and to allow postpublication open peer commentary (anonymous and onymous) to complement and supplement classical peer review.

And to end the arbitrary tyranny of journal impact factors (which just means the journal's average number of citations per article), let 1000 new Open Access metrics bloom -- a metric track-record, public and answerable.

Colquhoun, D. (2011) Publish-or-perish: Peer review and the corruption of science Guardian September 5 2011.

Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292.

Harnad, S. (1998) The invisible hand of peer review. Nature [Web Matters]

Harnad, S. (2003) Valedictory Editorial. Behavioral and Brain Sciences (Journal of Open Peer Commentary) 26(1): 1

Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1)

Lock, Stephen (1985) A difficult balance : editorial peer review in medicine London : Nuffield Provincial Hospitals Trust.

Megajournals, Quality Standards and Selectivity: Gaussian Facts of Life

nospam@example.com (Stevan Harnad) — Wed, 29 Jun 2011 16:36:50 +0100

SUMMARY: It is obvious that broad-spectrum, low-selectivity, pay-to-publish mega-journals -- whether Open Access or not Open Access -- can help meet many researchers' need to publish today, but it is certainly not true that that's the only way, the best way, or the most economical way to provide Open Access to their articles.

Like height, weight, and just about every other biological trait (including every field of human performance), scholarly/scientific quality is normally distributed (the "bell" curve), most of it around average, tapering toward the increasingly substandard in the lower tail of the bell curve and toward increasing excellence in the upper tail.

For some forms of human performance -- e.g., driving or doctoring -- we are satisfied with a pass/fail license cut-off.

For others, such as sports or musical performance, we prefer finer-grained levels, with a hierarchy of increasingly exacting -- hence selective -- performance standards.

But, as a matter of necessity with a finite (though growing) population and a bell curve with tapered tails, the proportion (and hence the number) of candidates and works that can meet higher and higher performance standards gets smaller and smaller.

Not only can everyone and everything not be in the top 10% or the top 1% or the top 0.1%, but because the bell curve's tail is tapered (it is a bell, not a pyramid), the proportion that can meet higher and higher standards shrinks even faster than a straight line.

Scholars and scientists' purpose in publishing in peer-reviewed journals -- indeed, the purpose of the "publish-or-perish" principle itself -- had always been two-fold: (1) to disseminate findings to potential users (i.e., mostly other scholars and scientists) and (2) to meet and mark a hierarchy of quality levels with each individual journal's name and its track-record for the rigor of its peer review standards (so users at different levels can decide what to read and trust and so quality can be assessed and rewarded by employers and funders).

In principle (though not yet in practice), journals are no longer needed for the first of these purposes, only the second -- but for that, they need to continue to be selective, ensuring that the hierarchy of quality standards continues to be met and marked.

It is obvious that broad-spectrum, low-selectivity, pay-to-publish mega-journals -- whether OA or not OA -- can help meet many researchers' need to publish today, but it is certainly not true that that's the only way, the best way, or the most economical way to provide OA for their articles:

Harnad, S. (2010) No-Fault Peer Review Charges: The Price of Selectivity Need Not Be Access Denied or Delayed. D-Lib Magazine 16 (7/8).

ABSTRACT: Plans by universities and research funders to pay the costs of Open Access Publishing ("Gold OA") are premature. Funds are short; 80% of journals (including virtually all the top journals) are still subscription-based, tying up the potential funds to pay for Gold OA; the asking price for Gold OA is still high; and there is concern that paying to publish may inflate acceptance rates and lower quality standards. What is needed now is for universities and funders to mandate OA self-archiving (of authors' final peer-reviewed drafts, immediately upon acceptance for publication) ("Green OA"). That will provide immediate OA; and if and when universal Green OA should go on to make subscriptions unsustainable (because users are satisfied with just the Green OA versions) that will in turn induce journals to cut costs (print edition, online edition, access-provision, archiving), downsize to just providing the service of peer review, and convert to the Gold OA cost-recovery model; meanwhile, the subscription cancellations will have released the funds to pay these residual service costs. The natural way to charge for the service of peer review then will be on a "no-fault basis," with the author's institution or funder paying for each round of refereeing, regardless of outcome (acceptance, revision/re-refereeing, or rejection). This will minimize cost while protecting against inflated acceptance rates and decline in quality standards.

Peer review itself, however, will, like homeostasis, always "defend" a level, whether that level is methodological soundness alone, methodological soundness and originality, methodological soundness, originality and importance, or what have you. The more exacting the standard, the fewer the papers that will be able to meet it. Perhaps the most important function of peer review is not the "marking" of a paper's having met the standard, but helping the paper to reach the standard, through referee feedback, adjudicated by an editor, sometimes involving several rounds of revision and re-refereeing.

Since peer review is an active, dynamical process of correction and improvement, it is not like the passive assignment of a letter grade to a finished work -- A, B, C, D. Rather, an author picks a journal that "defends" a target grade (A or B or C or D), submits the paper to that journal for refereeing, and then tries to improve the paper so as to meet the referees' recommendations (if any) by revising it.

There are, in other words, A, B, C and D journals, the A+ and A journals being the highest-standard and most selective ones, and hence the least numerous in terms of both titles and articles, for the Gaussian reasons described above.

A mega-journal, in contrast, is equivalent to one generic pass/fail grade (often in the hope that the "self-corrective" nature of science and scholarship will eventually take care of any further improvement and sorting that might be needed -- after publication, though "open peer review").

Maybe one day scholarly publication will move toward a model like that -- or maybe it won't (because users require more immediate quality markers, and/or because the post-publication marking is too uncertain and unreliable).

But what's needed today is open access to the peer-reviewed literature, published in A, B, C and D journals, such as it is, not to a pass/fail subset of it.

Hence pass/fail mega-journals are a potential supplement to the status quo, but not a substitute for it.

Stevan Harnad
EnablingOpenScholarship

Peer Review vs. Peer Ranking: Dynamic vs Passive Filtration

nospam@example.com (Stevan Harnad) — Wed, 16 Jun 2010 13:57:15 +0100

Chen & Konstan's (C & K) paper, "Conference Paper Selectivity and Impact" is interesting, though somewhat limited because it is based only on computer science and has fuller data on conference papers than on journal papers.

The finding is that papers from highly selective conferences are cited as much as (or even more than) papers from certain journals. (Journals of course also differ among themselves in quality and acceptance rates.)

Comparing the ceiling for citation counts for high- and low-selectivity conferences by analyzing only the equated top slice of the low-selectivity conferences, C & K found that that the high-selectivity conferences still did better, suggesting that the difference was not just selectivity (i.e., filtration) but also "reputation." (The effect broke down a bit in comparing the very highest and next-to-highest selectivity conferences, with the next-to-highest doing slightly better than the very highest; plenty of post hoc speculations were ready to account for this effect too: excess narrowness, distaste for competition, etc. at the very highest level, but not the next-highest)

Some of this smacks of over-interpretation of sparse data, but I'd like to point out two rather more general considerations that seem to have been overlooked or under-rated:

(1) Conference selectivity is not the same as journal selectivity: A conference accepts the top fraction of its submissions (whatever it sets the cut-off point to be), with no rounds of revision, resubmission and re-refereeing (or at most one cursory final round, when the conference is hardly in the position to change most of its decisions, since the conference looms and the program cannot be made much more sparse than planned). This is passive filtration. Journals do not work this way; they have periodic issues, which they must fill, but they can have a longstanding backlog of papers undergoing revision that are not published until and unless they have succeeded in meeting the referee reports' and editor's requirements. The result is that the accepted journal papers have been systematically improved ("dynamic filtration") through peer review (sometimes several rounds), whereas the conference papers have simply been statically ranked much as they were submitted. This is peer ranking, not peer review.

(2) "Reputation" really just means track record: How useful have papers in this venue been in the past? Reputation clearly depends on the reliability and validity of the selective process. But reliability and validity depend on more than the volume and cut-off point of raw submission rankings (passive filtration).

I normally only comment on open-access-related matters, so let me close by pointing out a connection:

There are three kinds of selectivity: journal selectivity, author selectivity and user selectivity. Journals (and conferences) select which papers to accept for publication; authors select which papers to submit, and which publication venue to submit them to; and users select which papers to read, use and cite. Hence citations are an outcome of a complex interaction of all three factors. The relevant entity for the user, however, is the paper, not the venue. Yes, the journal's reputation will play a role in the user's decision about whether to read a paper, just as the author's reputation will; and of course so will the title and topic. But the main determinant is the paper itself. And in order to read, use and cite a paper, you have to be able to access it. Accessibility trumps all the other factors: it is not a sufficient condition for citation, but it is certainly a necessary one.

Stevan Harnad
American Scientist Open Access Forum

PostGutenberg Peer Review

nospam@example.com (Stevan Harnad) — Wed, 12 May 2010 14:57:44 +0100

Joseph Esposito [JE] asks, in liblicense-l:

JE: What happens when the number of author-pays open access sites grows and these various services have to compete with one another to get the finest articles deposited in their respositories?

Green OA mandates require deposit in each author's own institutional repository. The hypothesis of Post-Green-OA subscription cancellations (which is only a hypothesis, though I think it will eventually prove to be right) is that the Green OA version will prove to be enough for users, leaving peer review as the only remaining essential publishing service a journal will need to perform.

Whether on the non-OA subscription model or on the Gold-OA author-pays model, the only way scholarly/scientific journals compete for content is through their peer-review standards: The higher-quality journals are the ones with more rigorous and selective criteria for acceptability. This is reflected in their track records for quality, including correlates of quality and impact such as citations, downloads and the many rich new metrics that the online and OA era will be generating.

JE: What will the cost of marketing to attract the best authors be?

It is not "marketing" but the journal's track record for quality standards and impact that attract authors and content in peer-reviewed research publication. Marketing is for subscribers (institutional and individual); for authors and their institutions it is standards and metrics that matter.

And, before someone raises the question: Yes, metrics can be manipulated and abused, in the short term, but cheating can also be detected, especially as deviations within a rich context of multiple metrics. Manipulating a single metric (e.g., robotically inflating download counts) is easy, but manipulating a battery of systematically intercorrelated metrics is not; and abusers can and will be named and shamed. In research and academia, this risk to track record and career is likely to counterbalance the temptation to cheat. (Let's not forget that metrics, like the content they are derived from, will be OA too...)

JE: I am not myself aware of any financial modeling that attempts to grapple with an environment where there are not a handful of such services but 200, 400, more.

There are already at least 25,000 such services (journals) now! There will be about the same number post-Green-OA.

The only thing that would change (on the hypothesis that universal Green OA will eventually make subscriptions unsustainable) is that the 25,000 post-Green-OA journals would only provide peer review: no more print edition, online edition, distribution, archiving, or marketing (other than each journal's quality track record itself, and its metrics). Gone too would be the costs of these obsolete products and services, and their marketing.

(Probably gone too will be the big-bucks era of journal-fleet publishing. Unlike with books, it has always been the individual journal's name and track record that has mattered to authors and their institutions and funders, not their fleet-publisher's imprimatur. Software for implementing peer review online will provide the requisite economy of scale at the individual journal level: no need to co-bundle a fleet of independent journals and fields under the same operational blanket.)

JE: As these services develop and authors seek the best one, what new investments will be necessary in such areas as information technology?

The best peer review is provided by the best peers (for free), applying the highest quality standards. OA metrics will grow and develop (independent of publishers), but peer review software is pretty trivial and probably already quite well developed (hence hopes of "patenting" new peer review "systems" are probably pipe-dreams.)

JE: Will the fixed costs of managing such a service rise along with greater demands by the most significant authors?

The journal quality hierarchy will remain pretty much as it is now, with the highest-quality (hence most selective) journals the fewest, at the top, grading down to the many average-level journals, and then the near-vanity press at the bottom (since just about everything eventually gets published somewhere, especially in the online era).

(I also think that "no-fault peer review" will evolve as a natural matter of course -- i.e., authors will pay a standard fee per round of peer review, independent of outcome: acceptance, revision/re-refereeing or rejection. So being rejected by a higher-level journal will not be a dead loss, if the author is ready to revise for a lower-level journal in response to the higher-level journal's review. Nor will rejected papers be an unfair burden, bundled into the fee of the authors of accepted papers.)

JE: As more services proliferate, what will the cost of submitting material on an author-pays basis be?

There will be no more new publishing services, apart from peer review (and possibly some copy-editing), and no more new journals either; 25,000 is probably enough already! And the cost per round of refereeing should not prove more than about $200.

JE: Will the need to attract the best authors drive prices down?

There will be no "need to attract the best authors," but the best journals will get them by maintaining the highest standards.

Since the peers review for free, the cost per round of refereeing is small and pretty fixed.

JE: If prices are driven down, is there any way for such a service to operate profitably as the costs of marketing and technology grow without attempting to increase in volume what is lost in margin?

Peer-reviewed journal publishing will no longer be big business; just a modest scholarly service, covering its costs.

JE: If such services must increase their volume, will there be inexorable pressure to lower some of the review standards in order to solicit more papers?

There will be no pressure to increase volume (why should there be)? Scholars try to meet the highest quality standards they can meet. Journals will try to maintain the highest quality standards they can maintain.

JE: What is the proper balance between the right fee for authors, the level of editorial scrutiny, and the overall scope of the service, as measured by the number of articles developed?

Much ado about little, here.

The one thing to remember is that there is a trade-off between quality-standards and volume: The more selective a journal, the smaller is the percentage of all articles in a field that will meet its quality standards. The "price" of higher selectivity is lower volume, but that is also the prize of peer-reviewed publishing: Journals aspire to high quality and authors aspire to be published in journals of high quality.

No-fault refereeing fees will help distribute the refereeing load (and cost) better than (as now) inflating the fees of accepted papers to cover the costs of rejected papers (rather like a shop-lifting surcharge!). Journals lower in the quality hierarchy will (as always) be more numerous, and will accept more papers, but authors are likely to continue to try a top-down strategy (as now), trying their luck with a higher-quality journal first.

There will no doubt be unrealistic submissions that can (as now) be summarily rejected without formal refereeing (or fee). The authors of papers that do merit full refereeing may elect to pay for refereeing by a higher-level journal, at the risk of rejection, but they can then use their referee reports to submit a more roadworthy version to a lower-level journal. With no-fault refereeing fees, both journals are paid for their costs, regardless of how many articles they actually accept for publication. (PotGutenberg publication means, I hasten to add, that accepted papers are certified with the name and track-record of the accepting journal, but those names just serve as the metadata for the Green OA version self-archived in the author's institutional repository.)

Harnad, S. (2009) The PostGutenberg Open Access Journal. In: Cope, B. & Phillips, A (Eds.) The Future of the Academic Journal. Chandos.

Harnad, S (2010) No-Fault Refereeing Fees: The Price of Selectivity Need Not Be Access Denied or Delayed. (Draft under refereeing).

And let's not forget what peer-reviewed research publishing is about, and for: It is not about provisioning a publishing industry but about providing a service to research, researchers, their institutions and their funders. Gutenberg-era publication costs meant that the Gutenberg publisher evolved, through no fault of its own, into the tail that wagged the paper-trained research pooch; in the PostGutenberg era, things will at last rescale into more proper and productive anatomic proportions...

Stevan Harnad
American Scientist Open Access Forum

Symptoms of Premature Gold OA -- and their Cure

nospam@example.com (Stevan Harnad) — Tue, 27 Apr 2010 11:16:26 +0100

"Gold" Open Access (OA) journals (especially high-quality, highly selective ones like PLOS Biology) were a useful proof of principle, but now there are far too many of them, and they are mostly not journals of high quality.

The reason is that new Gold OA journals are premature at this time. What is needed is more access to existing journals, not more journals. Everything already gets published somewhere in the existing journal quality hierarchy. The recent proliferation of lower-standard Gold OA journals arose out of the drive and rush to publish-or-perish, and pay-to-publish was an irresistible lure, both to authors and to publishers.

Meanwhile, authors have been sluggish about availing themselves of a cost-free way of providing OA for their published journal articles: "Green" OA self-archiving.

The simple and natural remedy for the sluggishness -- as well as the premature, low-standard Gold OA -- is now on the horizon: Green OA self-archiving mandates from authors' institutions and funders. Once Green OA prevails globally, we will have the much-needed access to existing journals for all would-be users, not just those whose institutions can afford to subscribe. That will remove all pretensions that the motivation for paying-to-publish in a Gold OA journal is to provide OA (rather than just to get published), since Green OA can be provided by authors by publishing in established journals, with their known track records for quality, and without having to pay extra -- while subscriptions continue to pay the costs of publishing.

If and when universal Green OA should eventually make subscriptions unsustainable -- because institutions cancel their subscriptions -- the established journals, with their known track records, can convert to the Gold OA cost-recovery model, downsizing to the provision of peer review alone (since access-provision and archiving will be done by the global network of Green OA Institutional Repositories), with the costs of peer review alone covered out of a fraction of the institutional subscription cancellation savings.

What will prevent pay-to-publish from causing quality standards to plummet under these conditions? It will not be pay-to-publish! It will be no-fault pay-to-be-peer-reviewed, regardless of whether the outcome is accept, revise, or reject. Authors will pay for each round of refereeing. And journals will (as now) form a (known) quality hierarchy, based on their track-record for peer-review standards and hence selectivity.

I'm preparing a paper on this now, provisionally entitled "No-Fault Refereeing Fees: The Price of Selectivity Need Not Be Access Denied or Delayed."

Stevan Harnad
American Scientist Open Access Forum

Don't Count Your Metric Chickens Before Your Open-Access Eggs Are Laid

nospam@example.com (Stevan Harnad) — Fri, 23 Oct 2009 11:42:19 +0100

In "Open Access is the short-sighted fight" Daniel Lamire [DL] writes:

DL: "(1) If scientific culture rewarded readership and impact above all else, we would not have to force authors toward Open Access."

(a) University hiring and performance evaluation committees do reward impact. (It is no longer true that only publications are counted: their citation impact is counted and rewarded too.)

(b) Soon readership (e.g., download counts, link counts, tags, comments) too will be counted among the metrics of impact, and rewarded -- but this will only become possible once the content itself is Open Access (OA), hence fully accessible online, its impact measurable and rewardable. (See references cited at the end of this commentary under heading METRICS.)

(c) OA mandates do not force authors toward OA -- or no moreso than the universal "publish or perish" mandates force authors toward doing and publishing research: What these mandates do is close the loop between research performance and its reward system.

(d) In the case of OA, it has taken a long time for the world scholarly and scientific community to become aware of the causal connection between OA and research impact (and its rewards), but awareness is at long last beginning to grow. (Stay tuned for the announcement of more empirical findings on the OA impact advantage later today, in honor of OA week.)

DL: "You know full well that many researchers are just happy to have the paper appear in a prestigious journal. They will not make any effort to make their work widely available because they are not rewarded for it. Publishing is enough to receive tenure, grants and promotions. And the reward system is what needs to be fixed."

This is already incorrect: Publishing is already not enough. Citations already count. OA mandates will simply make the causal contingency between access and impact, and between impact and employment/salary/promotion/funding/prizes more obvious and explicit to all. In other words, the reward system will be fixed (including the development and use of a rich and diverse new battery of OA metrics of impact) along with fixing the access system.

DL: "(2) I love peer review. My blog is peer reviewed. You are a peer and just reviewed my blog post."

Peer commentary is not peer review (as surely I -- who founded and edited for a quarter century a rather good peer-reviewed journal that also provided open peer commentary -- ought to be in a position to know!). Peer commentary (as well as post-hoc metrics themselves) are an increasingly important supplement to peer review, but they are themselves neither peer review nor a substitute for it. (Again, see references at the end of this commentary under the heading PEER REVIEW.)

DL: "(3) PLoS has different types of peer review where correctness is reviewed, but no prediction is made as to the perceived importance of the work. Let me quote them:"
Too often a journals decision to publish a paper is dominated by what the Editor/s think is interesting and will gain greater readership both of which are subjective judgments and lead to decisions which are frustrating and delay the publication of your work. PLoS ONE will rigorously peer-review your submissions and publish all papers that are judged to be technically sound. Judgments about the importance of any particular paper are then made after publication by the readership (who are the most qualified to determine what is of interest to them).

You have profoundly misunderstood this, Daniel:

(i) It is most definitely a part of peer review to evaluate (and where necessary correct) the quality, validity, rigor, originality, relevance, interest and importance of candidates for publication in the journal for which they are refereeing.

(ii) Journals differ in the level of their peer review standards (and with those standards co-vary their acceptance criteria, selectivity, acceptance rates -- and hence their quality and reliability).

(iii) PLoS Biology and PLoS Medicine were created explicitly in order to maintain the highest standards of peer review (with acceptance criteria selectivity and acceptance rates at the level of those of Nature and Science [which, by the way, are, like all peer judgments and all human judgment, fallible, but also corrigible post-hoc, thanks to the supplementary scrutiny of peer commentary and follow-up publications)).

(iv) PLoS ONE was created to cater for a lower level in the hierarchy of journal peer review standards. (There is no point citing the lower standards of mid-range journals in that pyramid as if they were representative of peer review itself.)

(vi) Some busy researchers need to know the quality level of a new piece of refereed research a-priori, at point of publication -- before they invest their scarce time in reading it, or, worse, their even scarcer and more precious research time and resources in trying to build upon it -- rather than waiting for months or years of post-hoc peer scrutiny or metrics to reveal it.

(v) Once again: commentary -- and, rarer, peer commentary -- is a supplement, not a substitute for peer review.

DL: "(4) Moreover, PLoS does publish non-peer-reviewed material, see PLoS Currents: Influenza for example."

And the journal hierarchy also includes unrefereed journals at the bottom of the pyramid. Users are quite capable of weighting publications by the quality track-record of their provenance, whether between journals, or between sections of the same journal. Caveat Emptor.

METRICS:

Brody, T., Kampa, S., Harnad, S., Carr, L. and Hitchcock, S. (2003) Digitometric Services for Open Archives Environments. In Proceedings of European Conference on Digital Libraries 2003, pp. 207-220, Trondheim, Norway.

Harnad, S. (2006) Online, Continuous, Metrics-Based Research Assessment. Technical Report, ECS, University of Southampton.

Brody, T., Carr, L., Harnad, S. and Swan, A. (2007) Time to Convert to Metrics. Research Fortnight pp. 17-18.

Brody, T., Carr, L., Gingras, Y., Hajjem, C., Harnad, S. and Swan, A. (2007) Incentivizing the Open Access Research Web: Publication-Archiving, Data-Archiving and Scientometrics. CTWatch Quarterly 3(3).

Harnad, S. (2008) Self-Archiving, Metrics and Mandates. Science Editor 31(2) 57-59

Harnad, S. (2008) Validating Research Performance Metrics Against Peer Rankings. Ethics in Science and Environmental Politics 8 (11) The Use And Misuse Of Bibliometric Indices In Evaluating Scholarly Performance

Harnad, S., Carr, L. and Gingras, Y. (2008) Maximizing Research Progress Through Open Access Mandates and Metrics. Liinc em Revista 4(2).

Harnad, S. (2009) Multiple metrics required to measure research performance. Nature (Correspondence) 457 (785) (12 February 2009)

Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1) Also in Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds. (2007)

Harnad, S; Carr, L; Swan, A; Sale, A & Bosc H. (2009) Maximizing and Measuring Research Impact Through University and Research-Funder Open-Access Self-Archiving Mandates. Wissenschaftsmanagement 15(4) 36-41

PEER REVIEW:

Harnad, S. (1978) BBS Inaugural Editorial. Behavioral and Brains Sciences 1(1)

Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in scientific quality control, New York: Cambridge University Press.

Harnad, S. (1984) Commentaries, opinions and the growth of scientific knowledge. American Psychologist 39: 1497 - 1498.

Harnad, Stevan (1985) Rational disagreement in peer review. Science, Technology and Human Values, 10 p.55-62.

Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A difficult balance: Peer review in biomedical publication.) Nature 322: 24 - 5.

Harnad, S. (1995) Interactive Cognition: Exploring the Potential of Electronic Quote/Commenting. In: B. Gorayska & J.L. Mey (Eds.) Cognitive Technology: In Search of a Humane Interface. Elsevier. Pp. 397-414.

Harnad, S. (1996) Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals. In: Peek, R. & Newby, G. (Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA: MIT Press. Pp 103-118.

Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292.

Harnad, S. (1998/2000/2004) The invisible hand of peer review. Nature [online] (5 Nov. 1998), Exploit Interactive 5 (2000): and in Shatz, B. (2004) (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield. Pp. 235-242.

Harnad, S. (2003/2004) Back to the Oral Tradition Through Skywriting at the Speed of Thought. Interdisciplines. Retour a la tradition orale: ecrire dans le ciel a la vitesse de la pensee. Dans: Salaun, Jean-Michel & Vendendorpe, Christian (dir). Le défi de la publication sur le web: hyperlectures, cybertextes et meta-editions. Presses de l'enssib.

Harnad, S. (2003) BBS Valedictory Editorial. Behavioral and Brain Sciences 26(1)

The Added Value of Providing Free Access to Paid-Up Content

nospam@example.com (Stevan Harnad) — Tue, 29 Sep 2009 10:03:05 +0100

On Sun, Sep 27, 2009 at 8:07 PM, Elizabeth E. Kirk, Dartmouth College Library, on liblicense-l, wrote:

"Stevan, it is, as you say, about content. But it's not only about the content of Dartmouth's research output, or that of our peers. It's also about the value of the content provided through publishers, and the willingness of readers and institutions to look for that value."

Elizabeth, I am not quite sure what you have in mind with the "it" that it "is... about." But if it's OA (Open Access), then the issue is not the value of the content or the contribution of the publisher or the willingness of readers and institutions to "look for" that value.

The value of peer-reviewed publication is already very explicitly enshrined in the fact that OA's specific target content is peer-reviewed content. What OA is equally explicitly seeking -- now that the advent of the online era has at last made it possible -- is free (online) access to that valued content, so it is no longer accessible only to those users whose institutions can afford to subscribe to the journal in which it was published, but to all would-be users, web-wide, in order to maximize research usage, impact and progress.

The cost of the portion of that value that is added by publishers is being paid in full by institutional subscriptions today. Hence what is missing is not a recognition of that value, but open access to that valued content.

That is why it is so urgent and important that each institution should first adopt a Green OA self-archiving mandate -- to make its own valued content openly accessible to all users web-wide and not just to those whose institutions can afford subscription access to the journals in which that content appears. This institutional self-help thereby also encourages reciprocal mandates by other institutions, to open the access to their own content as well.

Having thus seen to it that all its own peer-reviewed output is made (Green) OA, an institution is of course free to spend any spare cash it may have on paying for Gold OA publication, over and above what it already spends on subscriptions.

But an institution's committing pre-emptively to Gold OA funding compacts like COPE before or instead of mandating Green OA self-archiving is not only a waste of a lot of scarce money in exchange for very little OA value: it is also a failure to add OA value to all of the institution's research output at no extra cost (by mandating Green OA self-archiving).

"We both agree that the peer review process is a critical step in creating the finished work of scholarship, as well as "certifying" the work."

Yes indeed; but peer review is already being paid for -- in full, many times over -- for most journals today (including most of the journals users want and need most) through multi-institutional subscription fees, paid by those institutions that can afford to subscribe to any given journal. (There are about 10,000 universities and research institutions in all, worldwide, and 25,000 peer-reviewed journals, publishing about 2.5 million articles per year. No institution can afford to subscribe to more than a small fraction of those journals.)

To repeat: The value of peer review is not at issue. What is at issue is access -- access to paid-up, published, peer-reviewed articles.

"Currently, open access journals--as you rightly put it--are a very small subset of the publishing pie."

And committing to fund that small subset of an institution's own contribution to the "publishing pie" today, before or instead of committing to mandate OA for the vast supra-set of that institution's total journal article output, is committing to spend a lot of extra money for little OA while failing to provide a lot of OA for no extra money at all.

"Without a predictable financial stream, there are few avenues of growing an OA sector that can furnish peer review, copy editing, DOIs, and all of the other parts of publishing that have costs involved."

What is missing and urgently needed today -- for research and researchers -- is not "predictable financial streams" but online access to every piece of peer-reviewed research for every researcher whose institution cannot afford subscription access to it today. The "peer review, copy editing, DOIs, and all of the other parts of publishing that have costs involved" for those articles are already being paid in full today -- by the subscription fees of those institutions that can afford to subscribe to the journals in which they are published.

"Open Access" is about Access, not about "financial streams." The wide-open "avenue" that urgently needs to be taken today (for the sake of research and researchers today) is the already-constructed, and immediately traversable (green) toll-road to accessing the vast paid-up subscription stream that already exists today, not the uncertain and still-to-be-constructed (golden) road of "growing" a future "OA sector," by paying still more, over and above the tolls already being paid, for a new "stream" of Gold OA journals.

Institutions first need to provide immediate access to the peer-reviewed content they already produce today (its peer review already paid in full by subscriptions from all the institutions that can afford subscriptions to the journals in which that content already appears, today). Having done that, there's no harm at all in an institution's going on to invest its spare cash in growing new Gold OA "sectors."

But there's plenty of harm in doing so instead, pre-emptively, instead of providing the Green OA all institutions are already in the position to provide, cost-free, today.

"Trying to grow that kind of OA sector by supporting those costs, and overcoming the misconception that OA means "not peer reviewed" (which many people said about 10-15 years ago about all electronic journals, if you remember) is a honking good reason to join the compact."

Misconceptions about OA certainly abound. But the fact that OA means OA to peer-reviewed content has been stated explicitly from the very outset by the OA movement (BOAI), loud and clear for all those with ears to hear the honking. Committing to funding Gold OA for a small subset of an institution's peer-reviewed output instead of first mandating Green OA for the vast supra-set of an institution's peer-reviewed output is a rather pricey way to drive home the home-truth that OA's target content is indeed, and always has been, peer-reviewed content...

"That kind of OA sector, which of course can only be built when more institutions join us, is one that may create actual competition in journal publishing over time, by which I mean competition that results in lower prices, more players, and multiple models. It could include, as well, any current publisher who might wish to move to producer-pays from reader-pays."

"Prices, players, models, competition, payment, sectors": What has become of access -- access today, to today's peer-reviewed research -- in all this Gold Fever and "sector-growth" fervor, which seems to have left the pressing immediate needs of research and researchers by the wayside in favor of speculative future economics?

"We care very much about the stability of and access to our research."

Then why doesn't Dartmouth mandate Green OA self-archiving, today?

"We are working on that from a number of fronts and in multiple conversations. The compact is not our answer to everything. But we certainly won't step back from an opportunity to help create a more vibrant publishing landscape."

But why is committing to provide a little extra Gold OA for a small part of Dartmouth's peer-reviewed research output, at extra cost, being acted upon today, whereas committing to provide Green OA to all the rest of Dartmouth's peer-reviewed research output at no extra cost (by mandating Green OA) is still idling in "conversation" mode? -- especially since the cost of the value-added peer review for all the rest is already being paid in full by existing institutional subscriptions?

Stevan Harnad
American Scientist Open Access Forum

Post-Publication Metrics Versus Pre-Publication Peer Review

nospam@example.com (Stevan Harnad) — Thu, 23 Jul 2009 22:37:35 +0100

Patterson, Mark (2009) PLoS Journals measuring impact where it matters writes:

"[R]eaders tend to navigate directly to the articles that are relevant to them, regardless of the journal they were published in... [T]here is a strong skew in the distribution of citations within a journal typically, around 80% of the citations accrue to 20% of the articles... [W]hy then do researchers and their paymasters remain wedded to assessing individual articles by using a metric (the impact factor) that attempts to measure the average citations to a whole journal?

"Wed argue that its primarily because there has been no strong alternative. But now alternatives are beginning to emerge... focusing on articles rather than journals... [and] not confining article-level metrics to a single indicator... Citations can be counted more broadly, along with web usage, blog and media coverage, social bookmarks, expert/community comments and ratings, and so on...

"[J]udgements about impact and relevance can be left almost entirely to the period after publication. By peer-reviewing submissions purely for scientific rigour, ethical conduct and proper reporting before publication, articles can be assessed and published rapidly. Once articles have joined the published literature, the impact and relevance of the article can then be determined on the basis of the activity of the research community as a whole... [through] [a]rticle-level metrics and indicators..."

Merits of Metrics. Of course direct article and author citation counts are infinitely preferable to -- and more informative than -- just a journal average (the journal "impact factor").

And yes, multiple postpublication metrics will be a great help in navigating, evaluating and analyzing research influence, importance and impact.

But it is a great mistake to imagine that this implies that peer review can now be done on just a generic "pass/fail" basis.

Purpose of Peer Review. Not only is peer review dynamic and interactive -- improving papers before approving them for publication -- but the planet's 25,000 peer-reviewed journals differ not only in the subject matter they cover, but also, within a given subject matter, they differ (often quite substantially) in their respective quality standards and criteria.

It is extremely unrealistic (and would be highly dysfunctional, if it were ever made to come true) to suppose that these 25,000 journals are (or ought to be) flattened to provide a 0/1 pass/fail decision on publishability at some generic adequacy level, common to all refereed research.

Pass/Fail Versus Letter-Grades. Nor is it just a matter of switching all journals from assigning a generic pass/fail grade to assigning its own letter grade (A-, B+, etc.), despite the fact that that is effectively what the current system of multiple, independent peer-reviewed journals provides. For not only do journal peer-review standards and criteria differ, but the expertise of their respective "peers" differs too. Better journals have better and more exacting referees, exercising more rigorous peer review. (So the 25,000 peer-reviewed journals today cannot be thought of as one generic peer-review filter that accepts papers for publication in each field with grades between A+ and E; rather there are A+ journals, B- journals, etc.: each established journal has its own independent standards, to which its submissions are answerable)

Track Records and Quality Standards. And users know all this, from the established track records of the journals they consult as readers and publish in as authors. Whether or not we like to put it that way, this all boils down to selectivity across a gaussian distribution of research quality in each field. There are highly selective journals, that accept only the very best papers -- and even those often only after several rounds of rigorous refereeing, revision and re-refereeing. And there are less selective journals, that impose less exacting standards -- all the way down to the fuzzy pass/fail threshold that distinguishes "refereed" journals from journals whose standards are so low that they are virtually vanity-press journals.

Supplement Versus Substitute. This difference (and independence) among journals in terms of their quality standards is essential if peer-review is to serve as the quality enhancer and filter that it is intended to be. Of course the system is imperfect, and, for just that reason alone (amongst many others) a rich diversity of post-publication metrics are an invaluable supplement to peer review. But they are certainly no substitute for pre-publication peer review, or, most importantly, its quality triage.

Quality Distribution. So much research is published daily in most fields that on the basis of a generic 0/1 quality threshold, researchers simply cannot decide rationally or reliably what new research is worth the time and investment to read, use and try to build upon. Researchers and their work differ in quality too, and they are entitled to know a priori, as they do now, whether or not a newly published work has made the highest quality cut, rather than merely that it has met some default standards, after which users must wait for the multiple post-publication metrics to accumulate across time in order to be able to have a more nuanced quality assessment.

Rejection Rates. More nuanced sorting of new research is precisely what peer review is about, and for, and especially at the highest quality levels. Although authors (knowing the quality track-records of their journals) mostly self-select, submitting their papers to journals whose standards are roughly commensurate with their quality, the underlying correlate of a journal's refereeing quality standards is basically their relative rejection rate: What percentage of annual papers in their designated subject matter would meet their standards (if all were submitted to that journal, and the only constraint on acceptance were the quality level of the article, not how many articles the journal could manage to referee and publish per year)?

Quality Ranges. This independent standard-setting by journals effectively ranges the 25,000 titles along a rough letter-grade continuum within each field, and their "grades" are roughly known by authors and users, from the journals' track-records for quality.

Quality Differential. Making peer review generic and entrusting the rest to post-publication metrics would wipe out that differential quality information for new research, and force researchers at all levels to risk pot-luck with newly published research (until and unless enough time has elapsed to sort out the rest of the quality variance with post-publication metrics). Among other things, this would effectively slow down instead of speeding up research progress.

Turn-Around Time. Of course pre-publication peer review takes time too; but if its result is that it pre-sorts the quality of new publications in terms of known, reliable letter-grade standards (the journals' names and track-records), then it's time well spent. Offloading that dynamic pre-filtering function onto post-publication metrics, no matter how rich and plural, would greatly handicap research usability and progress, and especially at its all-important highest quality levels.

More Value From Post-Publication Metrics Does Not Entail Less Value From Pre-Publication Peer Review. It would be ironic if today's eminently valid and timely call for a wide and rich variety of post-publication metrics -- in place of just the unitary journal average (the "journal impact factor") -- were coupled with an ill-considered call for collapsing the planet's wide and rich variety of peer-reviewed journals and their respective independent, established quality levels onto some sort of global, generic pass/fail system.

Differential Quality Tags. There is an idea afoot that peer review is just some sort of generic pass/fail grade for "publishability," and that the rest is a matter of post-publication evaluation. I think this is incorrect, and represents a misunderstanding of the actual function that peer review is currently performing. It is not a 0/1, publishable/unpublishable threshold. There are many different quality levels, and they get more exacting and selective in the higher quality journals (which also have higher-quality and more exacting referees and refereeing). Users need these differential quality tags when they are trying to decide whether newly published work is worth taking the time to ready and making the effort and risk to try to build upon (at the quality level of their own work).

User/Author/Referee Experience. I think both authors and users have a good idea of the quality levels of the journals in their fields -- not from the journals' impact factors, but from their content, and their track-records for content. As users, researchers read articles in their journals; as authors they write for those journals, and revise for their referees; and as referees they referee for them. They know that all journals are not equal, and that "peer-reviewed" can be done at a whole range of quality levels.

Metrics As Substitutes for User/Author/Referee Experience? Is there any substitute for this direct experience with journals (as users, authors and referees) in order to know what their peer-reviewing standards and quality level are? There is nothing yet, and no one can say yet whether there will ever be metrics as accurate as having read, written and refereed for the journals in question. Metrics might eventually provide an approximation, though we don't yet know how close, and of course they only come after publication (well after).

Quality Lapses? Journal track records, user experiences, and peer review itself are certainly not infallible either, however; the usually-higher-quality journals may occasionally publish a lower-quality article, and vice versa. But on average, the quality of the current articles should correlate well with the quality of past articles. Whether judgements of quality from direct experience (as user/author/referee) will ever be matched or beaten by multiple metrics, I cannot say, but I am pretty sure they are not matched or beaten by the journal impact factor.

Regression on the Generic Mean? And even if multiple metrics do become as good a joint predictor of journal article quality as user experience, it does not follow that peer-review can then be reduced to generic pass/fail, with the rest sorted by metrics, because (1) metrics are journal-level, not article-level (though they can also be author-level) and, more important still, (2) if journal-differences are flattened to generic peer review, entrusting the rest to metrics, then the quality of articles themselves will fall, as rigorous peer review does not just assign articles a differential grade (via the journal's name and track-record), but it improves them, through revision and re-refereeing. More generic 0/1 peer review, with less individual quality variation among journals, would just generate quality regression on the mean.

REFERENCES

Bollen J, Van de Sompel H, Hagberg A, Chute R (2009) A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE 4(6): e6022. doi:10.1371/journal.pone.0006022

Brody, T., Harnad, S. and Carr, L. (2006) . Journal of the American Association for Information Science and Technology (JASIST) 57(8) pp. 1060-1072.

Garfield, E., (1955) Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas. Science 122: 108-111

Harnad, S. (1979) Creative disagreement. The Sciences 19: 18 - 20.

Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in scientific quality control, New York: Cambridge University Press.

Harnad, S. (1984) Commentaries, opinions and the growth of scientific knowledge. American Psychologist 39: 1497 - 1498.

Harnad, Stevan (1985) Rational disagreement in peer review. Science, Technology and Human Values, 10 p.55-62.

Harnad, S. (1990) Scholarly Skywriting and the Prepublication Continuum of Scientific Inquiry Psychological Science 1: 342 - 343 (reprinted in Current Contents 45: 9-13, November 11 1991).

Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A difficult balance: Peer review in biomedical publication.) Nature 322: 24 - 5.

Harnad, S. (1996) Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals. In: Peek, R. & Newby, G. (Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA: MIT Press. Pp 103-118.

Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292.

Harnad, S. (1998/2000/2004) The invisible hand of peer review. Nature [online] (5 Nov. 1998), Exploit Interactive 5 (2000): and in Shatz, B. (2004) (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield. Pp. 235-242.

Harnad, S. (2008) Validating Research Performance Metrics Against Peer Rankings. Ethics in Science and Environmental Politics 8 (11) Special Issue: The Use And Misuse Of Bibliometric Indices In Evaluating Scholarly Performance

Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1)

Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open Research Web: A Preview of the Optimal and the Inevitable, in Jacobs, N., Eds. Open Access: Key Strategic, Technical and Economic Aspects. Chandos.

OA Needs Open Evidence, Not Anonymous Innuendo

nospam@example.com (Stevan Harnad) — Mon, 08 Sep 2008 03:56:52 +0100

The testimony of "Ethan" regarding SJI's publishing practices could have been valuable -- if it had been provided non-anonymously. As it stands, it merely amounts to anonymous, nonspecific, unsubstantiated allegations. If those allegations against SJI are in reality true, then making them anonymously, as "Ethan" does, does more harm than good, because then they can be so easily discredited and dismissed, as being merely the anonymous, nonspecific, unsubstantiated allegations that they are. (If they are in reality false, then they are of course a depolorable smear.)

Richard Poynder is a distinguished and highly respected journalist and the de facto chronicler of the OA movement. I hope "Ethan" has contacted Richard, as he requested, giving him his real name, and the names of the SJI journal submissions that he refereed and recommended for rejection as having "zero scientific value." Richard can then fact-check (confidentially, without embarrassing the authors) whether or not any of those articles were published as such.

What “Ethan” should have done if he was, as he said, receiving articles of low quality to referee, in a “peer review” procedure of doubtful quality, was to resign as referee, request removal of his name from the list of referees — did “he”? and was his name removed? — and, if he felt strongly enough, offer to make his objective evidence available to those who may wish to investigate these publishing practices.

What is needed in order to expose shoddy publishing practices is objective, verifiable evidence and open answerability, not anonymous allegations (as in the entrails of Wikipedia, where pseudonymous bullying reigns supreme).

This is not, after all, whistle-blowing on the Mafia, that requires a witness protection program. If you offer to referee for a publisher with shoddy peer-review practices, you risk nothing if you provide what objective evidence you have of those practices.

I know that "publish or perish" has authors fearful of offending publishers by doing anything they think might reduce their chances of acceptance, and that referees often perform their gratis services out of the same superstitious worry; and I know that junior referees are worried about offending senior researchers if they are openly critical of their work, and that even peer colleagues and rivals are often leery of the consequences of openly dissing one another’s research.

Yet none of these bits of regrettable but understandable professional paranoia explain why "Ethan" felt the need to hide under a cloak of anonymity in providing objective evidence of dubious peer-review practices by a publisher and journals that hardly have the patina or clout of some of the more prestigious established publishers and journals.

Is it SJI's public threats of litigation, through postings like the one below, that have everyone so intimidated?

“Lies, fear and smear campaigns against SJI and other OA journals”

Surely the antidote against that sort of thing is open evidence, not anonymous innuendo. (Something better is needed by way of open evidence, however, than just contented testimonials elicited from accepted authors!)

Stevan Harnad
American Scientist Open Access Forum

Nature's Fall from Aside the Angels

nospam@example.com (Stevan Harnad) — Mon, 16 Jun 2008 13:11:29 +0100

Steve Inchcoombe, managing director of Nature Publishing Group writes, of Nature:

"We also support and encourage self-archiving of the authors final version of accepted articles."

But if you look in the Romeo directory of publisher self-archiving policies, you will find that whereas Nature is indeed among the 92% of journals that have endorsed the immediate self-archiving of the author's unrefereed first draft (the preprint), Nature is not among the 63% of journals that have endorsed the immediate self-archiving of the author's peer-reviewed final draft (the postprint) -- the one that is the real target of OA, and indispensable for research usage and progress.

Nature used to be "green" on the immediate self-archiving of both preprints and postprints, but, electing to take half of NIH's maximal allowable access embargo as its own minimum, Nature became one of the few journals that back-slid in 2005 to impose a 6-month embargo on open access to the peer-reviewed final draft .

It doesn't make much difference, because Institutional Repositories still have the almost-OA email eprint request-a-copy Button to tide over research usage needs during the embargo, but let it not be thought that Nature is still on the "side of the angels" insofar as OA is concerned...

Maxine Clarke, Publishing Executive Editor, Nature, replied: "Don't forget that people can always read the article in the journal, Stevan, as soon as it is published! The vast majority of scientists are either at an institution with a site license or can access the journal free via OARE, AGORA or HINARI, so they don't even have to take out a subscription."

But what about those would-be users worldwide who are "[n]either at an institution with a site license [n]or can access the journal free via OARE, AGORA or HINARI"? Is there any reason whatsoever why they should all be denied access for six months if they (or their institutions) do not "have [the funds] to take out a subscription"?

Because that, Maxine, is what OA is really all about.

Stevan Harnad
American Scientist Open Access Forum

RePEc, Peer Review, and Harvesting/Exporting from IRs to CRs

nospam@example.com (Stevan Harnad) — Wed, 31 Oct 2007 02:22:52 +0000

The new RePEc blog is a welcome addition to the blogosphere. The economics community is to be congratulated for its longstanding practise of self-archiving its pre-refereeing preprints and exporting them to RePEc.

Re: the current RePEc blog posting, "New Peer Review Systems":

Experiments on improving peer review are always welcome, but what the worldwide research community (in all disciplines, economics included) needs most urgently today is not peer review reform, but Open Access (OA) to its existing peer-reviewed journal literature. It's far easier to reform access than to reform the peer-review system, and it's also already obvious exactly what needs to be done and how, for OA -- mandate RePEc-style self-archiving, but for the refereed postprints, not just the unrefereed preprints -- whereas peer-review reforms are still in the testing stage. It's not even clear whether once most unrefereed preprints and all refereed postprints are OA anyone will still feel any need for radical peer review reform at all; it may simply be a matter of more efficient online implementation.

So if I were part of the RePEc community, I would be trying to persuade economists (who, happily, already have the preprint self-archiving habit) to extend their practise to postprints -- and to persuade their institutions and funders to mandate postprint self-archiving in each author's own OAI-compliant Institutional Repository (IR). From there, if and when desired, its metadata can then also be harvested by, or exported to, CRs (Central Repositories) like RePEc or PubMed Central. (One of the rationales for OAI-interoperability is harvestability.)

But the primary place to deposit one's own preprints and postprints, in all disciplines, is "at home," i.e., in one's own institutional archive, for visibility, usage, impact, record-keeping, monitoring, metrics, and assessment -- and in order to ensure a scaleable universal practise that systematically covers all research space, whether funded or unfunded, in all disciplines, single or multi, and at all institutions -- universities and research institutes. (For institutions that have not yet created an IR of their own -- even though the software is free and the installation is quick, easy, and cheap -- there are reliable interim CRs such as Depot to deposit in, and harvest back from, once your institution has its own IR.)

Stevan Harnad
American Scientist Open Access Forum

British Academy Report on Peer Review and Metrics

nospam@example.com (Stevan Harnad) — Tue, 04 Sep 2007 15:24:14 +0100

SUMMARY: (1) Peer review just means the assessment of research by qualified experts.
(2) Peer review, like all human judgment, is fallible, and susceptible to error and abuse.
(3) Funding and publishing without any assessment is not a solution:
(3a) Everything cannot be funded and even funded projects first need some expert advice in their design.
(3b) Everything does get published, eventually, but there is a hierarchy of journals with a hierarchy of peer-review quality standards, their names and track-records serving as an essential guide for users about what they can take the risk of trying to read, use and build upon.
(4) So far, nothing as good as or better than peer review (i.e., qualified experts vetting the work of their fellow-experts) has been found, tested and demonstrated.
(5) Peer review's efficiency can be improved in the online era.
(6) Metrics are not a substitute for peer review, they are a supplement to it.

"Help Wanted: A pall of gloom lies over the vital system of peer review. But the British Academy has some bright ideas". The Guardian, Jessica Shepherd reports, Tuesday September 4, 2007

The 4 Sept Guardian article on peer review (on the 5 Sept British Academy Report, to be published tomorrow) seems to be a good one. The only thing it lacks is some conclusions (which journalists are often reluctant to take the responsibility of making):

(1) Peer review just means the assessment of research by qualified experts. (In the case of research proposals, it is assessment for fundability, and in the case of research reports, it is assessment for publishability.)

(2) Yes, peer review, like all human judgment, is fallible, and susceptible to error and abuse.

(3) Funding and publishing without any assessment is not a solution:

(3a) Everything cannot be funded (there aren't enough funds), and even funded projects first need some expert advice in their design.

(3b) And everything does get published, eventually, but there is a hierarchy of journals, with a corresponding hierarchy of peer-review quality standards. Their names and track-records are essential for users, guiding them on what they can take the risk of trying to read, use and build upon. (There is not enough time to read everything, and it's to risky to try to build on just anything that purports to have been found -- and even accepted papers first need some expert advice in their revision.)

(4) So far, nothing as good as or better than peer review (i.e., qualified experts vetting the work of their fellow-experts) has been found, tested and demonstrated. So peer review remains the only straw afloat, if the alternative is not to be tossing a coin for funding, and publishing everything on a par.

(5) Peer review can be improved. The weak link is always the editor (or Board of Editors), who choose the reviewers and to whom the reviewers and authors are answerable; and the Funding Officer(s) or committee choosing the reviewers for proposals, and deciding how to act on the basis of the reviews. There are many possibilities for experimenting with ways to make this meta-review component more accurate, equitable, answerable, and efficient, especially now that we are in the online era.

(6) Metrics are not a substitute for peer review, they are a supplement to it.

In the case of the UK, a Dual Support System of prospective funding of (i) individual competitive proposals (RCUK) and (ii) retrospective top-sliced funding of entire university departments, based on their recent past research performance (RAE), metrics can help inform and guide funding officers, committees, editors, Boards and reviewers. And in the case of the RAE in particular, they can shoulder a lot of the former peer-review burden: The RAE, being a retrospective rather than a prospective exercise, can benefit from the prior publication peer review that the journals have already done for the submissions, rank the outcomes with metrics, and then only add expert judgment afterward, as a way of checking and fine-tuning the metric rankings. Funders and universities explicitly recognizing peer review performance as a metric would be a very good idea, both for the reviewers and the researchers being reviewed.

Harnad, S. (2007) Open Access Scientometrics and the UK Research Assessment Exercise. In Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds.

Brody, T., Carr, L., Gingras, Y., Hajjem, C., Harnad, S. and Swan, A. (2007) Incentivizing the Open Access Research Web: Publication-Archiving, Data-Archiving and Scientometrics. CTWatch Quarterly 3(3).

Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open Research Web: A Preview of the Optimal and the Inevitable, in Jacobs, N., Eds. Open Access: Key Strategic, Technical and Economic Aspects. Chandos.

Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in scientific quality control, New York: Cambridge University Press.

Harnad, Stevan (1985) Rational disagreement in peer review. Science, Technology and Human Values, 10 p.55-62.

Harnad, S. (1986) Policing the Paper Chase. [Review of S. Lock, A difficult balance: Peer review in biomedical publication.]Nature 322: 24 - 5.

Harnad, S. (1996) Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals. In: Peek, R. & Newby, G. (Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA: MIT Press. Pp 103-118.

Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292.

Harnad, S. (1998/2000/2004) The invisible hand of peer review. Nature [online] (5 Nov. 1998), Exploit Interactive 5 (2000): and in Shatz, B. (2004) (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield. Pp. 235-242.

Peer Review Reform Hypothesis-Testing (started 1999)

A Note of Caution About "Reforming the System" (2001)

Self-Selected Vetting vs. Peer Review: Supplement or Substitute? (2002)

Stevan Harnad
American Scientist Open Access Forum

Premature Rejection Slip

nospam@example.com (Stevan Harnad) — Tue, 17 Oct 2006 02:14:32 +0100

SUMMARY: Richard Poynder's essay on peer review is thoughtful and stimulating but quite wrong! Peer review is like water-quality control: Everyone shouldn't have to risk doing it all for himself (or relying on those who do it for themselves). (And it has nothing to do with OA, which is about making the filtered, quality-controlled water free for all.)

Richard Poynder, in "Open Access: death knell for peer review?" has written yet another thoughtful, stimulating essay. But I think he (and many of the scholars and scientists he cites) are quite baldly wrong on this one!

What is peer review? Nothing more nor less than qualified experts vetting the work of their fellow specialists to make sure it meets certain established standards of reliability, quality and usability -- standards that correspond to the quality level of the journal whose name and track-record certifies the outcome.

Peer review is dynamic and answerable: Dynamic, because it is not just an "admit/eject" decision by a gate-keeper or an "A/B/C/D" mark assigned by a schoolmarm, but an interactive process of analysis, criticism and revision that may take several rounds of successive revisions and re-refereeing. And answerable, because the outcome must meet the requirements set out by the referees as determined by the editor, sometimes resulting in an accepted final draft that is very different from the originally submitted preprint -- and sometimes in no accepted draft at all.

Oh, and like all exercises in human judgment, even expert judgment, peer review is fallible, and sometimes makes errors of both omission and commission (but neither machinery nor anarchy can do better). It is also approximate rather than exact; and, as noted, quality-standards differ from journal to journal, but are generally known from the journal's public track record. (The only thing that does resemble an A/B/C/D marking system is the journal-quality hierarchy itself: Meeting the quality-standards of the best journals is rather like receiving an A+, and the bottom rung is not much better than a vanity press.)

But here are some other home truths about peer review (from an editor of 25 years' standing who alas knows all too well whereof he speaks): Qualified referees are a scarce, over-harvested resource. It is hard to get them to agree to review, and hard to get them to do it within a reasonable amount of time. And it is not easy to find the right referees; ill-chosen referees (inexpert or biassed) can suppress a good paper or admit a bad one; they can miss detectable errors, or introduce gratuitous distortions.

Those who think spontaneous, self-appointed vetting can replace the systematic selectivity and answerability of peer review should first take on board the ineluctable fact of referee scarcity, reluctance and tardiness, even when importuned by a reputable editor, with at least the prospect that their efforts, though unpaid, will be heeded. (Now ask yourself the likelihood that that the right umpires will do their duty on their own, and not even sure of being heeded for their pains.)

Friends of self-policed vetting should also sample for a while the raw sludge that first makes its way to the editor's desk, and ask themselves whether they would rather everyone had to contend directly with that commodity for themselves, instead of having it filtered for them by peer review, as now. (Think of it more as a food-taster for the emperor at risk of being poisoned -- rather than as an elitist "gate-keeper" keeping the hoi-poloi out of the club -- for that is closer to what a busy researcher faces in trying to decide what work to risk some of his scarce reading time on, or (worse) his even scarcer and more precious research time in trying to build upon.)

And peer-review reformers or replacers should also reflect on whether they think that those who have nothing better to do with their time than to wade through this raw, unfiltered sludge on their own recognizance -- posting their take-it-or-leave-it "reviews" publicly, for authors and users to heed as they may or may not see fit -- are the ones they would like to trust to filter their daily sludge for them, instead of answerable editors' selected, answerable experts.

Or whether they would like to see the scholarly milestones, consisting of the official, certified, published, answerable versions, vanish in a sea of moving targets, consisting of successive versions of unknown quality, crisscrossed by a tangle of reviews, commentaries and opinions of equally unknown quality.

Not that all the extras cannot be had too, alongside the peer-reviewed milestones: In our online age, no gate-keeper is blocking the public posting of unrefereed preprints, self-appointed commentaries, revised drafts, and even updates and upgrades of the published milestones -- alongside the milestones themselves. What is at issue here is whether we can do without the filtered, certified milestones themselves (until we once again reinvent peer review).

The question has to be asked seriously; and if one hasn't the imagination to pose it from the standpoint of a researcher trying to make tractable use of the literature, let us pose it more luridly, from the standpoint of how to treat a family member who is seriously ill: navigate the sludge directly, to see for oneself what's on the market? ask one's intrepid physician to try to sort out the reliable cure from the surrounding ocean of quackery? And if you think this is not a fair question, do you really think science and scholarship are that much less important than curing sick kin?

Eppur, eppur... what tugs at me on odd days of the week is the undeniable fact that most research is not cited, nor worth citing, anyway, so why bother with peer review [or, horribile dictu, OA!] for all of that? And on the other end, the few authors of the very, very best work are virtually peerless, and can exchange their masterworks amongst themselves, as in Newton's day. So is all this peer review just to keep the drones busy? I can't say. (I used to mumble things in reply to the effect that "we need to countenance the milk in order to be ensured of the cream rising to the top" or "we need to admit the chaff in order to sift out its quota of wheat" or "we need to screen the gaussian noise if we want to ensure our ration of signal") [1], [2], [3]
Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A difficult balance: Peer review in biomedical publication.) Nature 322: 24 - 5.

But I can say that none of this has anything to do with Open Access, now (except that it can be obtruded, along with so many other irrelevant things, to slow OA's progress). If self-archiving mandates were adopted universally, all of this would be mooted. The current peer-reviewed literature, such as it is, would at long-last be OA -- which is the sole goal of the OA movement, and the end of the road for OA advocacy, the rest being about scholars and scientists making use of this newfound bonanza, whilst other processes (digital preservation, journal reform, copyright reform, peer review reform) proceed apace.

As it is, however, second-guessing the future course of peer review is still one of the at-least 34 pieces of irrelevance distracting us from getting round to doing the optimal and inevitable at long, long last...

    Harnad, S. (1990) Scholarly Skywriting and the Prepublication Continuum
of Scientific Inquiry. Psychological Science 1 342 - 343.

    Harnad, S. (1998) The invisible hand of peer review. Nature [online] (c. 5 Nov. 1998) Exploit Interactive version

    Peer Review Reform Hypothesis-Testing (started 1999)

    A Note of Caution About "Reforming the System" (2001)

    Self-Selected Vetting vs. Peer Review: Supplement or Substitute? (2002)

    Peer Review: Streamlining It vs. Sidelining It

Stevan Harnad
American Scientist Open Access Forum