Why Our Measures of Scholarly Prestige Are Anti-Intellectual and Random With Respect to Quality
Updated: 4 days ago
I should not be writing this! I absolutely have better things to do by almost any measure. Yet I have been asked to clarify which of my publications are “refereed” and which not. Unless I decide simply to resign, which would among other things leave my current students in the lurch, I cannot entirely refuse to respond.
It will be easier to respond at some length than to answer straightforwardly as requested— although I suppose I could flip a coin for all the grey cases, which include around half of my scholarly output over the years. I grant that people with other specializations may have other experiences, but I know what I am talking about for my area, and I doubt that my case is all that unusual.
Gimme Shelter
Let’s begin by recalling a few things about tenure. In addition to the tenure system being the actually existing structure that ensures basic job security and quality of life for teachers in academia—if there were actually-existing alternatives that did the same, we could discuss pros and cons of tenure in good faith—tenure was established in large part to give scholars a sort of shelter.
Behind this shelter, they had scope to engage in research, teaching, and writing that might not be considered legitimate as judged by certain outside-looking-in measures. For example, scholars might pursue inquiries that are inconvenient to religious orthodoxies, illegible or not-yet-proven in terms of reigning scholarly paradigms, or politically undesirable as judged by ruling regimes. In the case of Socrates, who sadly did not enjoy tenure, the charge was “corrupting the minds of youth.” Quality teaching in our classrooms, if allowed to unfold as it should—with both teachers who have scope to challenge students and students who have space to explore cultural critiques—is hard to sustain without such shelter.
One of the pleasures of gaining tenure is that for years I have often been able to disregard—not without some risk, perhaps vaguely like Socrates but with exceedingly lower stakes—the forced, misleading, counterproductive, anti-intellectual, and all-around bogus hard distinction between “refereed” and “not-refereed.” (To repeat: all these insults might not apply in every field, but I insist that I know about my own.) My choices have unfolded partly on principle but mainly as an evolving result of “following my nose” toward the best ways I could find to nurture and disseminate my research, teaching, and outreach.
My Most-Cited Article: Productive, Irrelevant, or Counter-Productive?
One example is my most-cited article, an analysis of Madonna that appeared in an edited book on religion and popular culture. It was polished and edited more extensively than almost any of my publications. But was it “refereed” (read: legible to bean-counters as “productivity”)? I have never been certain how anyone who evaluated me answered this. In retrospect this article probably reads to most colleagues as being at least in the ballpark of “real scholarship”—although articles that are book chapters are not supposed to fully count as “articles”—since this book is from a top university press and has been widely cited. However, in real time this was ambiguous. Key colleagues thought I was wasting my time and were at least tempted to lump the work as “not-scholarly” and to discount it when assessing me: not refereed, slightly embarrassing, and a terrible choice as measured in career-building opportunity costs.
I also know all the following things about this piece. Its first draft was not part of my core scholarly agenda—it was intended as a quick side project urging the Christian Century magazine not to be dismissive of a Madonna video. Likely because their editors were dismissive, just as I feared, this manuscript was rejected. (Let’s belabor it, I’m evoking what I said earlier about needing shelter to venture beyond what is legible in reigning scholarly paradigms.) So I expanded it into a “normal” article with footnotes, which garnered another rejection from a journal of feminist theology. Thus this project failed at both public popularization and refereed productivity. But I am thrilled that it wound up in the book where it did (refereed or otherwise) because by the time I had rewritten it, with excellent editing, it was far better and far more widely read.
Eight Factors of Quality and Value That Are Not Measured by Refereeing
In my experience, complexities like the above are the rule, not the exception. Let's state the problem in general terms before I return with more examples. There are continua between all the following poles:
Writing of high intellectual quality—or not.
Writing that represents lengthy time investments and serious innovation—or not.
Journals with editorial boards (or functional equivalents) that are distinguished and/or full of interlocutors one should desire—or not.
Journals (or equivalents) that extensively vet and edit drafts of submitted work—or not. This may be called "refereeing"—or not. "Refereed journals" by no means always edit.
Journals (or equivalents) that have readerships that one would desire as dialogue partners—or not.
Journals or equivalents with crossover readerships between academia and various publics—or not. This can be important, or not. In fields where I work it is important.
Publications with long word/page counts—or not. This is a huge factor because it takes more work, resulting in higher quality and value, to make something longer into something shorter. If I double the time I spend on an article to cut it in half and polish its prose, I get a fourfold negative incentive for this trouble: no doubling of “assessed productivity” but rather a 50% cut.
Projects undertaken in part for synergies with quality teaching, for example to build breadth of expertise or think through problems in one's syllabi—or not. Prioritizing such things strikes me as a clear virtue but rarely a good careerist move.
Not a single one of these continua corresponds to “refereed/not-refereed!” Every single one includes a great deal of refereed work at the undesirable end of its continuum. Every single one includes a great deal of not-refereed work or ambiguous cases at the positive end.
Common wisdom in our university—structuring the logic that determines who has a viable career—posits that a refereed-or-not continuum is a valid approximate composite measure of quality/value on these fronts. I am explicitly denying that this is true. No doubt some scholars’ mileage will vary, but in my experience with dozens of journals, the distinction “refereed-or-not-refereed” is an independent variable that is close to random with respect to the above eight continua.
A great deal of what I read that wastes my time—exhibiting low quality, low time investment by authors, low readership, non-optimum word counts (sometimes grossly inflated due to a lack of editing, sometimes overly compressed), and low value for teaching—is refereed. A great deal of what I read that has high quality, high investment… let’s just say “worthwhile” and worth emulating—is either explicitly “not refereed” or we circle back to a coin flip methodology for deciding ambiguous cases.
To summarize, a tenured scholar with the prerogative and responsibility of exercising judgment should very often prefer the non-refereed.
Further Concrete Examples
Since I really do feel aggrieved to repeat these points—I’ve been making them for years, long enough to suspect that they will be ruled out of bounds and treated as an annoying distraction—I will offer only a few more examples with fewer details.
Several of my most important and prestigious articles took the form of longish review articles in widely read journals. These journals would be deemed at first approximation “refereed”—or if not, then toward the valorized end of a “quality/prestige” continuum. I understand my contributions as short articles with pointed scholarly interventions—often involving a great deal of effort to make them fit into shortish "review article" slots. On balance, I worked harder on these and am prouder of them compared to many of my refereeed articles.
Bean-counters ask: aren’t these “mere reviews” (read: we will discount them if we count them at all)? Well, yes. Are they scholarly articles? Yes! Are they in "refereed journals?" (Generally probably yes, but now I add that some cases are clearer than others, and the quality, word counts, and dissemination are comparable across the board.) Are they "refereed articles" in these journals? (Time for the coin flip; it is unclear how to categorize invited review essays, except that it involves discounting.)
After we sort all this out, we can turn to other of my review articles—similar in quality and sometimes more widely read by more desirable interlocutors—but now placed in journals that rank lower on a prestige continuum.
The Yo-Yo Effect
This is an enormous bloc of work that is like a yo-yo—on the way up it is “real scholarship,” on the way down “he is doing mere reviews,” then up again to “real,” down to “non-refereed,” and so on.
Likewise we can play yo-yo with dozens of shorter reviews—clearly not articles but an enormous body of work that I defend fiercely as a scholarly contribution.
What about invited work for reference works or encyclopedias? I have written some such pieces that were quick and easy, either written for the occasion or spun out of my books. However, also in this category are several of my longest, most ambitious, most time-consuming—and in one case most prestigious—works on my resume. Pieces along this entire spectrum have often been treated by bean-counters roughly the same.
I agreed to write several of these because I conceived them as building blocks for books. One such article is lengthy and not “encyclopedia-like” in form. Nevertheless it will continue to be rated a “mere encyclopedia article” (read: surely discounted, possibly ignored or even treated as a negative) until it becomes the heart of an introduction to a refereed book (the holy grail!). “Objectively” this will have been a counter-productive waste of time until the book comes out—except that this is part of my process for writing the book.
Throwing Bad Energy After Good
So far I have been talking about throwing good energy after bad, as measured by the university’s incentive system. But there also is throwing bad energy after good, whether for benign motives or to game the system.
Some of my least important and least read articles—representing my smallest time investment and least value added—were refereed. When my university insulted its faculty by imposing a one-size-fits-all, staggeringly ugly, and outrageously time-wasting straitjacket of a website for reporting productivity—with an apparent goal of making it easier for administrators to create quantitative reports—they touted the website as a time saving benefit to faculty. Supposedly it could search all relevant databases and import data automatically. Guess what its search found for me, from more than 100 articles and reviews? An exceedingly small number drawn entirely from this category of my least important work.
When a Day Counts More than a Year and Readers Are on the Internet
It is not an exaggeration, but a straight literal fact, that I have seen lines on my resume that only cost me a couple afternoons of work being weighted by bean-counters “the same”—or even more!—than resume lines that cost me years of effort. The most egregious cases hinged on whether to count things as refereed. On one side were the above-noted “spin it out in an afternoon” articles that landed in a refereed column. On the other side were my two most ambitious book chapters—synthesizing massive amounts of research. It would have cost me less trouble to expand these latter two chapters into entire books than it did to cut their first drafts down to article length. Yet both can be categorized as “mere invited articles” in edited volumes which might be construed (read: discounted) as reference works.
Turning to on-line journals that are increasingly central to scholarly discussion—with boards that may or may not “referee” depending on semantics—similar themes tumble through similar ambiguities. I’ve written many short and simple online pieces. But I also I have written many long online essays that will be building blocks for my books.
All this is before we turn to this blog—refereed by me and me alone!
Which of these on-line efforts are “refereed scholarship”? I predict that my deans’ numerical printouts will reflect zero of them—although there is no clarity about it and every reason to assume that that definitions will be applied inconsistently. Nevertheless, some such online pieces have been vetted, edited, and read more—not less—than my comparable pieces in standard (“legacy?”) journals. Some are on par in quality with the best things I have ever written, even though I exercised my judgment to place them online.
Exercising Judgment When “Refereed” is Random With Respect to Value
Such exercise of judgment—whether about publishing online, accepting invitations to write for edited books, spending time on “mere reviews,” etc.—is not solely my prerogative as a tenured scholar. It is my responsibility as a scholar in a land-grant public university.
So let’s restate what I said at the start. As a tenured scholar, it is part of my intellectual and spiritual discipline to try (in most cases) to bracket and disregard the forced, misleading, counterproductive, anti-intellectual, and all-around bogus hard distinction between the “refereed” and the “not-refereed” in fields where I work.
I will continue to aim for a composite quality score on all the above-mentioned continua—amid which the refereed vs. not-refereed distinction may not reduce entirely to a random variable, but comes close.
