The Israeli Framework for the Assessment of Legal Research: A Possible Alternative/Supplement to REF?

Evaluating academic research output is a never-ending challenge. Such evaluation is an integral part of internal hiring, promotion, and tenure procedures, and of external funding decisions and institutional rankings. The proper way to evaluate academic publications has been the subject of fierce debate across disciplines, including law. Publications in business law have generally been subjected to the same yardsticks as other legal publications, but they may also be evaluated as a distinct sub-discipline of business studies, as the quality assessment initiative of the Australian Business Deans’ Council demonstrates.

The traditional method for academic evaluation is specific review of each scholarly work, assessing its originality, rigor, significance, etc, in light of relevant disciplinary criteria. This method, known as ‘peer review’, is still the dominant model of assessment by academic journals and funding institutions. It is also used in various research quality assessment systems, most notably the UK’s Research Evaluation Framework (REF). However, peer review is rather costly, at times impractical, and might be subjective and biased. 

These concerns have generated an increased interest in the use of quantitative indicators for research evaluation. Impact factor (IF), the mean citation counts of items published in journals in the preceding two years, has become particularly salient in this context. IF and other metrics are influencing promotion decisions, grant allocations, and project appraisals. In the same vein, various quantitative measures have been developed to assess academic institutions. 

Notwithstanding its scientific allure, the use of quantitative measures to assess research has been heavily criticized by the academic community. Critics noted various shortcomings of reliance on quantitative indicators. First and foremost, complete reliance on such indicators loses sight of the intrinsic value of academic work. Second, different fields have distinct ‘citation communities’, so it makes no sense to use IF as a universal benchmark. Third, the use of quantitative measures might create perverse incentives that undermine scientific innovation and reward mediocre work. Additional criticism is more specific. For example, it may take a while for a cutting-edge article to gain citations; and niche studies might garner only few citations regardless of their quality. 

The distinctive characteristics of the legal academic domain pose three additional challenges. First, the universe of English-language legal periodicals is unique in that it consists of two categories of publications, with different review policies. One category is that of conventional peer-reviewed journals. Most journals in this category follow a double-blind system, in which the identities of both the authors and the reviewers are concealed from each other throughout the review process, and publication decisions are made by senior scholars in the field. The other category, unique to the legal field, consists of student-edited journals, which are usually affiliated with law schools and managed by the students of these schools; most of these journals are affiliated with U.S. law schools. In the latter category, student-editors decide which articles to accept and reject without a formal peer review process. A second challenge is the absence of a comprehensive, universally endorsed, quantitative ranking of law journals. The major academic rankings—JCR by Clarivate analytics (WoS), Journal metrics by Scopus (CiteScore), SCImago Journal Rankings (SJR), and Washington & Lee Law Journal Rankings are based on distinct databases and different methodologies, and it is difficult to compare them or to integrate their results. Finally, in non-English speaking jurisdictions, such as Israel, scholars tend to publish in both their local language and English, in domestic and foreign law journals.

In late 2016, the Deans of the four Israeli law schools operating within public universities appointed Michael Birnhack (Tel Aviv University), Oren Perez (Bar Ilan University), Ronen Perry (University of Haifa), and Doron Teichman (Hebrew University in Jerusalem) as members of the Inter-University Committee for the Evaluation and Ranking of Legal Publications. The Committee was entrusted with the task of producing a coherent ranking of legal publications that (1) does not rely exclusively on quantitative measures, thereby alleviating some of the abovementioned criticism, and (2) responds to the distinctive characteristics of legal publishing. It reviewed over 900 publications, and suggested a four-tier categorization, accompanied by relative weights, which are detailed in its recently published Report.

The Committee classified publications into four categories: A*, A, B, and C, in accordance with other ranking models. The division between these categories takes a pyramidal form. Category A* was intended to include the top 5% of ranked journals, and ultimately includes 4.3% of ranked journals (39/908) and less than 2% of all legal periodicals (that is, when unranked journals are taken into account). This group of elite publication outlets includes American journals such as the Harvard Law Review and the Yale Law Journal, as well as British journals, such as the Oxford Journal of Legal Studies and the Modern Law Review. Category A was intended to include the next 15% of ranked journals, and eventually consists of 16% (146/908). The two highest categories together comprise one fifth of ranked journals. Category B was intended to include the next 35% of ranked journals, and ultimately covers a little more than 34% (310/908). Category C was intended to include the remaining 45%, and includes slightly less than 46% (413/908). In fact, there is a fifth category or layer consisting of thousands of unranked journals.

The Pyramidal Division of Law Journals

The Report was approved by the faculty councils of the four constituent law schools in April and May 2019, and is likely to be incorporated into public funding decisions in the near future. While the Report is intended to serve the Israeli academic community, the Committee believes that it may be of interest to other legal academics, as the problems it tackles transcend national boundaries.

Ronen Perry is a Professor of Law and Director of the Aptowitzer Center for Risk, Liability, and Insurance at the University of Haifa.