Reverse detail from Kakelbont MS 1, a fifteenth-century French Psalter. This image is in the public domain. Daniel Paul O'Donnell

Forward to Navigation

Cædmon Citation Network - Week 4

Posted: Jun 11, 2016 10:06;
Last Modified: Jun 11, 2016 11:06

---

Hello!

This blog comes to you a day later than usual, as Friday’s work ended up taking a lot longer than I thought and I ran out of time! To be honest, this week was spent much like last week: checking our Zotero bibliography against other bibliographies of Cædmon scholarship.

I ended up re-doing a bit of my work from last week, as I learned in my meeting with Dan on Monday that our scope was a bit wider than I had previously thought. I was worried that I had not been considering certain entries in the various bibliographies to be “about Cædmon enough”, so I decided to go through the entries again and add some that I may have missed. It makes sense to add more rather than less, as I can simply remove an article from the list if I read it and realise it has nothing to do with Cædmon. At the moment our bibliography is almost complete, and we have nearly 700 entries!

What are we going to do with this giant list of articles and books? Well, firstly I have to acquire access to each entry, either via JSTOR, inter-library loans, or through one of our library’s other databases. Then I read through EVERYTHING and count each quote and mention of Cædmon and note which of the approximately sixty different editions of the Hymn are cited. We have also decided to try and note every other citation as well. For example if one article about “Cædmon’s Hymn” cites a book about the history of peanut butter sandwiches, I will take note of it, as there may be other pieces of Cædmon scholarship that also cite that book about the history of peanut butter sandwiches. It will be interesting to see if there are identifiable relationships between writing about Cædmon and seemingly unrelated topics – not peanut-butter-sandwich-history obviously, I just haven’t eaten breakfast yet so I am giving you a delicious example.

How am I going to keep track of all this? Good question! We will need a database that I can use to mark down each citation as I come across them in my reading. On Monday Dan and I discussed at length what we will need from this database, and how we would like it to work. At first we were hoping something on Google Forms would do the trick for us, however we discovered as we talked that we need more control over our information than this tool would allow.

One problem emerged when we realised that among our gigantic list of 700 articles (and books, etc) we would find certain works that were actually editions of the Hymn not included in our original list of editions. We would need a way to add this piece to the Editions list… Several other concerns were raised as well, but to be honest I am finding them difficult to explain without drawing you all a little picture. (I should ask Dan how to add images to these blog posts!)

I mentioned at some point that I would pick the brain of my boyfriend, Garret Johnson, who has his degree in Computer Science from the University of Lethbridge and is my go-to person whenever I have a question about these sorts of things. Dan suggested that he could hire Garret to build our database if he would be willing, as someone with a programming background could probably produce what we need a lot faster than either Dan or I working on it ourselves. So that is our current plan! Garret will begin building us a database that will suit our needs and my job for next week will be to start acquiring the 700 articles and books on our list. By the end of next week I am sure I will have thoroughly annoyed the librarians at school with the amount of inter-library loans I will be requesting.

Until next week!

Colleen

----  

Cædmon Citation Network - The Return

Posted: May 19, 2016 10:05;
Last Modified: May 19, 2016 11:05

---

Hello, Readers of Dan’s Blog!

My name is Colleen Copland, and I am a student of Dan’s who will be working with him on the Cædmon Citation Network which he and Rachel Hanks began work on last summer. I will be blogging here weekly, and thought I’d use this first post to introduce myself and more-or-less explain the project as I understand it so far. I am still familiarizing myself with everything, so my descriptions may fall short of the actual scope of the project or they might be totally off-base altogether, but as I learn more I will let you know all the juicy details!

Little intro on myself: I am an undergraduate student at the University of Lethbridge, majoring in English and hoping to be accepted into the English/Language Arts Education program this fall (cross your fingers for me, internet!). I have taken three courses with Dan in the past two years, Medieval English, Intro to Old English, and Advanced Old English in which we spent an entire semester reading Beowulf. Suffice to say I think Dan is a pretty excellent prof and I am excited to work for him this summer so I can continue to learn from him!

The Cædmon Citation Network (also known as the Cædmon Bibliography Project and possibly a few other names – I will need to ask Dan if there is something he’d like me to call it officially) is a gathering of data on the citations of various editions of Cædmon’s Hymn. The project is interested in tracking how long it takes a new edition of a work to start being cited in studies of said work. Cædmon’s Hymn, since it is such a short piece, has been re-translated and re-published a great many times since 1644, which should allow us to notice some patterns in the way each new edition is cited.

The project is also interested in looking at the differences between the citing of digital editions of works as opposed to print editions. Many people assume that it takes longer for digital editions to begin being cited, but this project aims to suggest that they are actually cited more quickly. It will be interesting to see what the data shows us.

Where are we right now with regards to the project? Personally, I am becoming oriented with the project’s goals and working to gain access to all of the excellent data collected by Rachel Hanks who worked on the project last year – figuring out where everything was left off and where Dan would like it to go this summer.

I am excited about gathering more information and will share it with you as I progress. It often seems that I gain a better understanding of a project when I explain what is happening to someone else, so I think this blog will be an excellent tool. It will also serve as a good record of what went on at different points during the project for Dan and I. Any questions you might have can be left in the comments section that I believe is located below this post…

Until next week,

Colleen

----  

First thing we do, let's kill all the authors. On subverting an outmoded tradition (Force2015 talk)

Posted: Mar 01, 2015 17:03;
Last Modified: Oct 01, 2015 15:10

---

This is a rough approximation (with some esprit d’escalier) of my speaking script from my talk at the “Credit where Credit is Due”: session at Force2015, January 13, 2015. We were asked to be controversial, so I tried to oblige.

Contents

Introduction

I’m not sure that this paper is going to introduce anything new to the discussion of authorship issues, perhaps just raise some reminders couched in the terms and methodology of a discipline that is only beginning to grapple with problems natural scientists have had to deal with for years. I’m also not going to solve anything, but rather walk through the origins of the problem and propose some potential avenues for change. But I’m also not going to be discussing tweaks or improvements to the system. Instead, I’m going to be arguing that our current author attribution system for scholarly and scientific publications is fundamentally broken and that the only route forward is sabotage.

Would we create scientific authors if they didn’t already exist?

The question I’d like to begin with is the following:

“If we didn’t have the concept of the scientific and scholarly author, would we create it?”

The answer, I think, is that we would not.

The International Council of Medical Journal Editors’ definition of authorship vs. a traditional dictionary definition

This is because what we currently describe as a scientific author actually looks nothing like almost anything else we would describe using the term “author”—as you can see if we compare the definition of scientific authorship as described by the International Committee of Medical Journal Editors and a relatively standard definition of regular authorship taken from an online dictionary:

A typical dictionary definition: Author, n., A writer of a book, article, or document.


ICMJE definition The ICMJE recommends that authorship be based on the following 4 criteria:

* Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND

* Drafting the work or revising it critically for important intellectual content; AND

* Final approval of the version to be published; AND

* Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

….

Contributors who meet fewer than all 4 of the above criteria for authorship should not be listed as authors, but they should be acknowledged. Examples of activities that alone (without other contributions) do not qualify a contributor for authorship are acquisition of funding; general supervision of a research group or general administrative support; and writing assistance, technical editing, language editing, and proofreading. Those whose contributions do not justify authorship may be acknowledged individually or together as a group under a single heading (e.g. “Clinical Investigators” or “Participating Investigators”), and their contributions should be specified (e.g., “served as scientific advisors,” “critically reviewed the study proposal,” “collected data,” “provided and cared for study patients”, “participated in writing or technical editing of the manuscript”). (Emphasis added).

In other words, while in the world outside scholarly and scientific communication, we normally think of the author as the person who actually does the writing, in the the world of research communication, it is entirely possible to have writers who are not authors and authors who are not writers. And that, it seems to me, means we are fairly deep down the rabbit hole.

The nature of the problem

There has been a lot of excellent work why our definition of authorship in research communication is the way it is, by Michel Foucault, by Roger Chartier, Mario Biagioli, Mark Rose, and others (see especially Biagioli and Galison, eds., Scientific Authorship: Credit and Intellectual Property in Science). They have tied it to issues of authority, early intellectual property rights, aesthetics, and economics.

I like to think, however, that the problem really comes down to four main issues:

The inertia of words

The first major problem with scientific authorship, in my view at least, is that our practical definition is changing faster than our term’s connotative implications.

That is to say, while it is entirely possible for us to adapt and bend the term “author” to match our current scientific practice—even if that scientific practice results in such abnormal beasts as the “writer-who-is-not-an-author” and the “author-who-is-not-a-writer”—we cannot as easily let go of our term’s original connotations. Somewhere in the back of our minds, we still believe that authors should be writers, even if our heads and contemporary practice tell us that this is simply neither reasonable nor practical for our biggest projects in the age of Big Science.

We can see that this is so, indeed, if we read through the rest of the ICMJE definition, to get to the bit where they discuss how their definition of authorship should not be abused in order to unreasonably exclude participants who deserve credit for authorship by denying them opportunities to participate in the writing and editing of the article:

These authorship criteria are intended to reserve the status of authorship for those who deserve credit and can take responsibility for the work. The criteria are not intended for use as a means to disqualify colleagues from authorship who otherwise meet authorship criteria by denying them the opportunity to meet criterion #s 2 or 3. Therefore, all individuals who meet the first criterion should have the opportunity to participate in the review, drafting, and final approval of the manuscript. (Emphasis added).

There are two things significant about this proviso. The first is the nature of the abuse that the ICMJE is attempting to prevent: the case of somebody being denied authorship credit on the basis of the second and third criteria (i.e. because they were prevented from particiating in the drafting of the article or were not given a veto over its contents) despite the fact that they met the requirements of the first and fourth criteria (i.e. made “substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work” and agreed to be held “accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved”). In other words, the ICMJE is worried that “the writing” might be used as a technicality to deny authorship to those who made otherwise major contributions to the science on which this writing reports.

But the second thing about this proviso is that it doesn’t protect against the opposite eventuality—that is to say that somebody who did participate in “the writing” might be unfairly denied authorship credit because they were prevented from making substantial contributions to the design or performance of the research or because they were not allowed to claim responsibility for the work. In other words, the ICMJE does not (for obvious reasons) think that preventing somebody from “designing the experiment” might be used as a technicality to deny somebody scientific credit. Or again in other words: while the ICMJE is prepared to accept that somebody could be deserving of authorship if they were unfairly denied access to the writing and editing, they don’t think the same thing about somebody whose only participation in the article was, well, “authorship” in the sense that everybody but academics understand the word. In scientific authorship, writing is a technicality in a way that participation in the actual experimental design and conduct is not.

The conservatism of aesthetics

This brings me to the second cause of our problems with the concept in research communication: the conservative nature of aesthetics. Because the connotations of the word are so strong, even if our practice has bent the definition almost completely out of shape, we also have a strong aesthetic feeling for where authorship attribution has to go in a scientific article: on the “byline,” between the title and the abstract or introduction. Indeed, a surprising amount of criticism of (and resistance to) “author inflation” rests on the simple idea that it looks ridiculous if you have eight hundred or a thousand authors between the title and the abstract of a scientific article—something that has affected even our opening Keynote speaker, Chris Lintott, in his attempts to accurately credit the profession and citizen-scientists who participated in his experiments with crowdsourcing.

Economic utility

The third reason why it has proven so difficult to let go of the idea that a scientific author must be a “writer” in some way has to do with economic utility. As the great historians of this topic have demonstrated, the decline in anonymous authorship came about in large part through the efforts of booksellers and publishers to create a mechanism for asserting copyright. If a work is anonymous, then it belongs to nobody (or everybody). If it has an author, then this author can alienate their rights to a publisher in exchange for money and their identity can be used subsequently both to brand the works in question (i.e. “another play from the great Shakespeare”) and identify fraudulent reproductions.

Although the situation in scholarship is not exactly analogous, the basic idea still pertains: naming the author of scientific works allows us to use such work as a mechanism for calculating economic value and reward and assigning scientific responsibility. In the specific case of academic researchers, authorship become something could count and use to make comparisons between individual researchers (your worth, of course, rises with the number and prestigiousness of your articles) and something you could use to certify against fraudulent science.

Scalability

And finally, there is the issue of scalability. There have always been epistemological differences between creative/literary and research/scientific authorship. The one is an act of creation (you create rather than discover poems), while the other is an act of discovery (good scientists discover rather than create their results). But the importance of these differences was obscured when both types of authors collaborated in similar sized groups (i.e. of one or two at most). In the age of single author science and scholarship, it was easy to see an equivalence between “doing the science” and “being the writer of the paper” since both activities were usually performed by the same person and because individual experiments could be completely reported in at most a few individual papers.

But this equivalence does not scale. As experiments got bigger and more complex, as projects began to involve many more experiments and participants, and as individual papers increasingly began to report on smaller fragments of the total scientific output of a project, this rough equivalence between “person who wrote the paper” and “people who did the science” became increasingly untenable and you end up with problems like that the ICMJE proviso quoted above is trying to head off—the case of participants in a project being denied access to the byline of an article solely because they weren’t given a change to wrangle a few words in the article reporting on their work.

Why words can hurt us

The point of all this is to show that the real cause of the “authorship crisis” is not the ever-increasing number of authors, but the fact that we are using the wrong word to describe what these people are and identify the quality we are trying to capture. And, unfortunately, that we are using a word that brings with it a lot of connotative and conceptual baggage that we simply cannot easily get rid of. While the term “author” (with all its connotative history) makes sense as a way of describing creative writers, for whom the actual act of composition is their primary activity, it simply does not work as a way of describing scientific authorship, for whom the act of composition is, ultimately, secondary to the research that precedes it. Poetry does not exist unless it is expressed in some form and its expression is in some sense at least coincidental with its composition (you can’t think of a poem without thinking of the words you will use to tell others about it). But while science ultimately requires communication, this communication cannot occur without prior activity: you can (and indeed probably should) do scientific research before you have decided on the precise words you are going to use to report on your results.

(And as a brief aside here, it is worth noting that science is not the only field of endeavour in which writing is secondary: the same is true, for example, in the legal or policy worlds where the actual writing is less important than the goals behind it, except that, in contrast to scientific credit systems, we don’t have any problem in these worlds in distinguishing between those who develop and those who merely draft legal bills or policy documents—there is no rule in parliament or congress that says that MPs or Senators can only be listed as the authors of a bill if they participated in its drafting).

So what to do?

This brings us to the problem of what to do. If the term “author” is bringing with it too strong a set of connotations to allow us to accurately capture what we want to capture (which is presumably participation in science rather than participation in typing), what can we do to change the current situation?

Accept that we don’t actually care who wrote the article

The first thing we need to do is accept that our current concept of scientific authorship is both an abuse of the term and brings with it far too much unhelpful baggage. That is to say, we need to recognise that we don’t actually care all that much about who wrote the scientific articles we read—beyond perhaps in the limited sense of making sure that those who did write out the results are rewarded for that writing. What we are actually trying to capture with our current credit/reward system is not participation in writing, but participation in communicated science.

This recognition is important, if for nothing else, in that it should free us of our aesthetic objections to long author lists. As long as we think that scientific authorship is actually about writing, then aesthetic objections to long authorship lists remain at least somewhat valid: it is indeed absurd to think that 800 people could possibly be responsible for the composition of a 6 page physics article. But if we stop thinking that what we are trying to capture is who wrote the article instead of who did the science reported on in the article, then the aesthetic objection becomes far less significant: if the people being credited are not actually authors, then we can stop thinking of their names as belonging on the byline; or we can stop thinking that the “byline” on a scientific article is in any way analogous to the byline on a newspaper article or novel.

Recognise that “authorship” is really just a special form of acknowledgement

Once we accept that scientific authorship systems are not actually about who wrote the article, it becomes easier to understand the next conceptual adjustment we need to make: recognising that “authorship” in a scientific sense is really just a special form of acknowledgement—or, in more concrete terms, that the “byline” in an article is really just an arbitrary, privileged, “above the fold” extension of the acknowledgements section.

You can see this if you compare the case of scientific authorship against that of poetry. Both books of poetry and scientific articles name authors and, commonly, have an acknowledgements section. The difference, however, is that where there is a clear epistemological difference between those mentioned in the byline and acknowledgements section in a book of poetry, there is (despite many attempts to develop one) no such clear demarcation in a scientific article. In a book of poetry, one will often find the author acknowledging the help of librarians who aided them in finding specific reference works, friends who hosted them while they were writing, thank yous to their agents and fellow poets for support, and so on. While these are all arguably people who helped the poet do his or her work, there is still a pretty clear distinction between helping a poet compose and actually composing yourself: nobody thinks the bartender at the poet’s favorite watering hole is actually an acknowledged coauthor of the poetry itself (well, not seriously, at least).

The people acknowledged in a scientific article, however, are, for the most part, those specifically responsible for conducting the science upon which the article is reporting: the people who did the calculations, who designed or ran the apparatus, who built the detectors, and so on. These are generally not people who had a purely social connection to the scientific work in the article but instead were directly responsible for its results. Our hypothetical poet would probably still have been able to compose poetry without the assistance of his or her agent. A scientific “author” would have nothing to write about if it were not for the people who helped make the discoveries in the first place.

This means, in turn, that the authorship-acknowledgements distinction in a scientific article is different from the similarly-named distinction in literary contexts. In contrast to the quite concrete distinction between the “person who composed the work” and “people who assisted the composer” we find in a literary work, in a scientific work, the distinction between “named as author” and “acknowledged as helper” is far more arbitrary, despite attempts such as those of the ICMJE to come up with discriminators. Instead of being able to make a clear binary distinction between those who have primary responsibility for a piece of science and those who merely assisted (as we are, in many ways, able to do in the case of literary authorship), what we are really doing in the case of scientific authors is attempting to determine the point on a scale of participation at which we decide to stop rewarding those who participated in our research. People to the left of the line we draw get their names put in the byline and are able to use the article on their CVs; people to the right of it get our best wishes, but little else, from their participation.

Understand that this conceptual problem is not amenable to tinkering

Since the problem with scientific authorship is conceptual—i.e. we are using the wrong criteria in attempting to determine credit—it is also not amenable to tinkering around the edges. Once you accept that an author might be somebody who doesn’t write and that a writer might not be an author, you are far beyond the power of any definitional tweak to save your system. Since the problem is the fact that we maintain an arbitrary distinction between those acknowledged on the “byline” and those acknowledged in the “acknowledgements,” reforms that improve the granularity of either without addressing the fundamental problem that we make the distinction at all is going to fail. Such reforms are attempts at refining the accounting for responsibility for “the article,” when what we really need is a system that recognises both that “the article” is only a second subcomponent of the scientific endeavour and that it is participation in reported science, not participation in the reporting of science, that our reward systems should be attempting to capture.

In fact, the only solution, in the end, is to stop using “authorship” as the primary measure of scientific participation. In the age of Big Science, the article is a better indication of the success of a project than the success of any individual within that project. We will only solve the issue of credit when we stop privileging authorship over participation.

Realise that there is no opportunity for external change

Although the problem is not amenable to tinkering, it is also true that it is not amenable to fiat. Because so much currently rides on authorship credit, we will find it almost certainly impossible to change formally in a top-down fashion. As advocates of Open Access discovered in the early years, change only comes when there is a critical mass that is comfortable with the innovation, but a critical mass only develops when the change itself is already understood to work within the current system. As various people have pointed out, Academia is very much a prestige economy and prestige markers are extremely resistant to change: scientists may want to publish in Open Access journals, but they need to publish in prestigious ones—and prestige seems, in large measure, to be a function of time and familiarity.

This is where the “sabotage” comes in. If you can’t change a system externally, then the only option left is to change it from within. And if the problem that we are facing with our current authorship systems is that they force us to make arbitrary distinctions among participants, then the solution is to refuse to make those distinctions. Since scientific authorship measures the wrong thing and excludes people who should be credited as participants solely on the relatively arbitrary grounds of whether they participated in the drafting of the article, then the solution is to stop using “writing” as a criterion for authorship: in other words, move the line that distinguishes acknowledgements above the fold from those below to put all of the people whose participation made the science possible above. It is only when the byline becomes indistinguishable from the acknowledgements section that the system will have been modified to the point where we can begin to work on more granular systems of identifying (and rewarding) actual scientific participation. Because, as Syndrome argues in the The Incredibles, “when everyone’s super, no one is!”

Conclusion

“Sabotage” is a strong word, but we are actually facing a pretty fundamental problem with our current attribution system. While equating “authorship” with “scientific productivity” made rough sense in the age of single-scientist experiments (and still does, to a large extent, in the current age of single scholar humanities research), the concept simply does not scale. It is difficult to apply to even moderately large collaborative teams and it is simply impossible to apply to the gigantic teams responsible for today’s biggest projects.

The reason for this, however, is that the concept is simply wrong. When we count authorship on scientific papers as part of our evaluation of an individual researcher, we are actually counting the wrong thing. We do not, on the whole, actually care that much whether a given scientist wrote the papers involved. What we are really attempting to capture is how productive and effective that scientist is as a participant in the science that is reflected in those papers—i.e. in the communication and discovery of communicable results. This does not mean that the article itself is irrelevant to science—you can’t have science without the communication of results. But it does mean that authorship of papers (authorship in the sense of “writing”) is no longer an adequate metric of scientific participation. The PI who conceptualised the project, the researchers who designed the equipment or methods, the people who calculated and reported the results—all of these are necessary to the production of good science whether or not they participated in actually typing or editing the articles in which this science is reported. Systems that fail to recognise this, such as that of the ICMJE with its fetishisation of, in essence, typing, are ultimately not going to solve a fundamental problem that has to do with the very term we use to describe the metric.

The answer to my question at the beginning is that we would not create the concept of the scientific author as a credit metric if it did not already exist. Now that it is causing serious trouble, it is time to kill it off.

----  

When everyone’s super… On gaming the system

Posted: May 23, 2012 20:05;
Last Modified: May 23, 2012 21:05

---

note: first published on the dpod blog

Syndrome: Oh, I’m real. Real enough to defeat you! And I did it without your precious gifts, your oh-so-special powers. I’ll give them heroics. I’ll give them the most spectacular heroics the world has ever seen! And when I’m old and I’ve had my fun, I’ll sell my inventions so that everyone can have powers. Everyone can be super! And when everyone’s super… [chuckles evilly] no one will be.

The Incredibles

Here’s a funny little story about how a highly specialised journal gamed journal impact measurements:

The Swiss journal Folia Phoniatrica et Logopaedica has a good reputation among voice researchers but, with an impact factor of 0.655 in 2007, publication in it was unlikely to bring honour or grant money to the authors’ institutions.

Now two investigators, one Dutch and one Czech, have taken on the system and fought back. They published a paper called ‘Reaction of Folia Phoniatrica et Logopaedica on the current trend of impact factor measures’ (H. K. Schutte and J. G. Švec Folia Phoniatr. Logo.59, 281–285; 2007). This cited all the papers published in the journal in the previous two years. As ‘impact factor’ is defined as the number of citations to articles in a journal in the past two years, divided by the total number of papers published in that journal over the same period, their strategy dramatically increased Folia‘s impact factor this year to 1.439.

In the ‘rehabilitation’ category, shared with 26 other journals, Folia jumped from position 22 to position 13.

—“Tomáš Opatrný. Playing the system to give low-impact journal more clout:. Nature 455, 167 (11 September 2008).

Assessing (and hence demonstrating) impact is a difficult but important problem in contemporary academia.

For most of the last century, university researchers have been evaluated on their ability to “write something and get it into print… ‘publish or perish’” (as Logan Wilson put it as early as 1942 in The Academic Man: A Study in the Sociology of a Profession, one of the first print citations of the term).

As you might expect, the development of a reward system built on publication led to a general increase in number of publications. Studies of science publication suggest a growth rate in the number of scientific articles and journals of between 2 and 5% per year since 1907 (a rate that leads to doubling roughly every 15 years). There is also evidence for a particularly marked rise in numbers after the 1950s.

This kind of growth vitiates the original point of the metric. If everybody publishes all the time, then the simple fact of publication is no longer sufficient as a proxy for excellence. You could count the sheer number of publications—a measure that is in fact widely used in popular contexts to imply productivity—were it not so obviously open to abuse: unless you institute some kind of control over the type and quality of publication, a system that simply counts publications will lead inevitably to an increase in number, and a corresponding decrease in quality, originality, and length.

It is perhaps for this reason that modern peer review systems begin to be institutionalised in the course of the second half of the last century. In fact, while peer review is probably understood to be the sine qua non of university research, and while it is possible to trace sporadic examples of activity resembling peer review back into the classical period, peer review in its modern form in fact really only begins to take shape only in the period from the 1940s-1970s. Major scientific journals, including Science and The Journal of the American Medical Association, for example, begin to make systematic use of external reviewers only in the 1940s, partially as an apparent response to the growing number and specialisation of submissions.

As you might expect, the peer review/reward system has itself been gamed. In the same way a reward system built on counting publications leads inevitably to an increase in the number of publications, a reward system build on counting peer-reviewed publications leads, inevitably, to an increase in the number of peer-reviewed publications… and the size and number of the journals that publish them.

Journal impact measurements are a controversial response to the not-surprising fact that peer review has also become an insufficient proxy for excellence. It is still relatively early days in this area (though less so in the natural sciences) and there is as yet not a complete consensus as to how impact should be quantified. As a result, the measures can still take many forms, from lists of ranked journals, to citation counts, to circulation and aggregation statistics, to in the case of on-line journals even more difficult-to-interpret statistics such as bounce and exit rates.

Regardless of how the impact factor debate settles out, however, it is only a matter of time until it too is gamed. Indeed, as the example of Folia Phoniatrica et Logopaedica suggests, it even may not be a matter of time. If you count citations, researchers will start ensuring they get cited. If you rank journals, they will ensure their journals fit your ranking criteria. If you privilege aggregation, the aggregators will be flooded with candidates for aggregation. And it is not clear that commercial understandings of good web analytics are really appropriate for scholarly and scientific publishing.

But the Folia Phoniatrica et Logopaedica example is also interesting because I’m not sure it is a bad thing. I can’t independently assess Opatrný’s claim that the journal is well respected though faring badly in impact measurements, but it wouldn’t surprise me if he was right. And the fact that a single researcher in a single article was able to more than double his journal’s impact score simply by citing every paper published in the journal in the previous two years leaves me… quite happy for him. I doubt there are many people who would consider the article cited by Opatrný to be in some way fraudulent. Instead, I suspect most of us consider it evidence (at best) that there are still some bugs in the system and (at worst) of a successful reductio ad absurdum–similar in a certain sense to Alan Sokol’s submission to Social Text.

None of this means that impact metrics are an intrinsically bad thing. Or that peer review isn’t good. Or that researchers shouldn’t be expected to publish. In fact, in many ways, the introduction of these various metrics, and the emphasis they receive in academia, is very good. Peer review has become almost fully institutionalised in the humanities in the course of my career. When I was a graduate student in the early 1990s, most journals I submitted to did not have formal explanation of their review policies and many were probably not, strictly speaking, peer reviewed. But it was difficult to tell and nobody I knew even attempted to distinguish publications on their CVs on the basis of whether or not they were peer reviewed. We were taught to distinguish publications (and the primary metric was still number of publications) on the basis of genre: you separated reviews from encyclopedia entries from notes from lengthy articles. A review didn’t count for much, even if we could have shown it was peer reviewed, and a lengthy article in what “everybody knew” to be a top journal counted for a lot, whether it was peer reviewed or not.

By the time I was department chair, 10 years later, faculty members were presenting me with CVs that distinguished output on the basis of peer review status. In these cases, genre was less important that peer review status. Reviews that were peer-reviewed were listed above articles that weren’t and journals began being quite explicit about their reviewing policies. The journal I helped found, Digital Medievalist, began from its first issue with what we described as “ostentatious peer review”: we named the referees who recommended acceptance on every article, partially as a way of borrowing their prestige for what we thought was, at the time, a fairly daring experiment in open access publication.

But we did this also because we thought (and think) that peer review is a good thing. My peer reviewed articles are, in almost every case, without a doubt better written and especially better and more carefully argued than my non-peer-reviewed articles. I’ve had stupid comments from referees (though none as stupid as seems to be the norm on grant applications), but there is only one case I can think of where I really couldn’t see how satisfying what the referee wanted wouldn’t improve things.

And the same is true for publication frequency. On the whole, my experience is that people who publish more (within a given discipline) also tend to publish better. I don’t publish too badly for somebody in my discipline. But most of the people who publish more than me in that same discipline are people I’d like to emulate. It is possible to game publication frequency; but on the whole, even the people who (I think) game it are among our most productive and most interesting scholars anyway: they’d still be interesting and productive even if they weren’t good at spinning material for one article into three.

So what does it mean that Schutte and Švec were able to game the impact measure of their journal with such apparent ease? And what should we say in response to the great uproar (much of it in my view well-founded) about the introduction of journal ranking lists by the ESF and Australian governments in recent years? Obviously some journals simply are better than others–more prestigious, better edited, more influential, containing more important papers. And it is difficult to see how frequency of citation is a bad thing, even if its absence is not necessarily evidence something is not good or not important. I would still rather have a heavily cited article in the PMLA than an article nobody read in a journal nobody has ever heard of.

Perhaps the most important thing is that it suggests, as Barbossa says to Miss Turner in Pirates of the Caribbean concerning the “Pirates’ Code,” that these kind of metrics should really be considered “more what you’d call ‘guidelines’ than actual rules.” Journals (and articles), that have a high impact factor, lots of citations, and are heavily read, are probably to be celebrated. But impact, citations, and subscription are not in themselves sufficient proxies for quality: we should expect to find equally good articles, journals, and scholars to exist with lower numbers in all these areas. And more importantly, we should expect to find that any quantifiable criteria we do establish will almost immediately be gamed by researchers in the field: most people with PhD-level research positions got where they are, after all, because they were pretty good at producing what examiners wanted to hear.

The real issue, then, is that metrics like “impact” or “peer review” or even “quantity” are attempts to use quantitative values as substitutes for qualitative assessment. The only real way of assessing quality is through qualitative assessment: that is to say by assessing a work on its own merits in relation to the goals it sets itself in terms of audience, impact, and subject matter, including the reasonableness of these goals. An article by an author who is not famous, in an obscure field, in a on-line journal that has no subscribers, and is not frequently cited may or may not represent poor quality work–in much the same way as might a frequently cited article in a popular field in a journal that is published by a famous academic, in the journal of the main scholarly society in a discipline. What is (or should be) important to the assessor is how reasonably each author has defined his or her goals and how well the resulting work has done in relation to those goals.

And this is where academics’ ability to game any other system becomes a virtue. Since there is no single metric we can create that researchers as a group will not figure out how to exploit (and then in short order), we should accept that we will simply never be able to propose a quantitative measurement for assessing intrinsic quality. What we can rely on, however, is that researchers will, on the whole, try to present their work in its best light. By asking the researchers to explain how their work can be best assessed, and being willing to evaluate that both that explanation and the degree to which the work meets the proposed criteria, we can find a way of comparatively evaluating excellence. Journals, articles, and researchers, that define, then meet or exceed reasonable targets for their disciplines and types of work, are excellent. Those that don’t, aren’t.

And in the meantime, we’ll develop far more innovative measurements of quality.

----  

Extracting a catalogue of element names from a collection of XML documents using XSLT 2.0

Posted: Sep 15, 2011 17:09;
Last Modified: May 23, 2012 19:05

---

We are trying to build a single stylesheet to work with the documents of two independent journals. In order to get a sense of the work involved, we wanted to create a catalogue of all elements used in the published articles. This means loading as input document directories’ worth of files and then going through extracting and sorting the elements across all the input documents.

Here’s the stylesheet that did it for us. It is probably not maximally optimised, but it currently does what we need. Any suggestions for improvements would be gratefully received.

Some notes:

  1. Our goal was to pre-build some templates for use in a stylesheet, so we formatted the elements names into xsl templates.
  2. Although you need to use this sheet with an input document, the input document is not actually transformed (the files we are extracting the element names from are loaded using the collection() function). So it doesn’t matter what the input document is as long as it is valid XML (we used the stylesheet itself)
<?xml version="1.0"?> 
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

<!-- this output is because we are going to construct 
ready-made templates for each element -->
    <xsl:output method="text"/>

<!-- for pretty printing -->
    <xsl:variable name="newline">
        <xsl:text> 
        </xsl:text>
    </xsl:variable>

<!-- Load the files 
in the relevant directories -->
    <xsl:variable name="allFiles"
    select="collection(iri-to-uri('file:///some/path/to/the/directories?select=*.xml;
    recurse=yes'))"/>

<!-- Dump their content into a single big pile -->
    <xsl:variable name="everything">
        <xsl:copy-of select="$allFiles"/>
    </xsl:variable>

<!-- Build a key for all elements using their name -->
    <xsl:key name="elements" match="*" use="name()"/>

<!-- Match the root node of the input document
(since the files we are actually working on have been 
loaded using the using the collection() function, nothing 
is actually going to happen to this element) -->
    <xsl:template match="/">

       <!-- this is information required to turn the output into an 
              XSL stylesheet itself -->
        <xsl:text>&lt;xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
            version="1.0"></xsl:text>
        <xsl:value-of select="$newline"/>
        <xsl:text>&lt;!--Summary of Elements --&gt;</xsl:text>
        <xsl:value-of select="$newline"/>
        <xsl:value-of select="$newline"/>

       <!-- this invokes the collection of all elements in all the files
       in the directory for further processing -->
        <xsl:for-each select="$everything">

           <!-- This makes sure we are dealing with the first named key -->
            <xsl:for-each 
            select="//*[generate-id(.)=generate-id(key('elements',name())[1])]">

               <!-- sort them -->
                <xsl:sort select="name()"/>

                <xsl:for-each select="key('elements', name())">

                   <!-- this makes sure that only the first instance 
                    of each element name is outputted -->
                    <xsl:if test="position()=1">
                        <xsl:text>&lt;xsl:template match="</xsl:text>
                        <xsl:value-of select="name()"/>
                        <xsl:text>"> </xsl:text>
                        <xsl:value-of select="$newline"/>
                        <xsl:text>&lt;!--</xsl:text>
                        <!-- this counts the remaining occurences -->
                        <xsl:value-of select="count(//*[name()=name(current())])"/>
                        <xsl:text> occurences</xsl:text>
                        <xsl:text>--&gt;</xsl:text>
                        <xsl:value-of select="$newline"/>
                        <xsl:text>&lt;/xsl:template></xsl:text>
                        <xsl:value-of select="$newline"/>
                        <xsl:value-of select="$newline"/>
                    </xsl:if>
                </xsl:for-each>
            </xsl:for-each>
        </xsl:for-each>
        <xsl:value-of select="$newline"/>
        <xsl:text>&lt;/xsl:stylesheet></xsl:text>
    </xsl:template>
</xsl:stylesheet>
----  

Why should I write for your Wiki? Towards an economics of collaborative scholarship.

Posted: Dec 15, 2006 17:12;
Last Modified: Jan 04, 2017 16:01

---

Originally presented at the conference of the Renaissance of America. San Francisco, CA. March, 2006.

I’d like to begin today by telling you the story of how I came to write this paper. Ever since I was in high school, I have used a process called “constructive procrastination” to get things done. This system involves collecting a bunch of projects due at various times and then avoiding work on the one that is due right now by finishing something else instead. Or as my wife, who actually teaches this system says: “if you want me to get your project done today, give me something more important to avoid working on.”

In this particular case, the important thing I wanted to avoid doing was this lecture. And the thing I did instead in order to avoid it was work on an article for the Wikipedia. Or rather—and to be honest, worse—work on revising an article I put up on the Wikipedia almost a year ago when I was was trying to avoid working on an article on Fonts for the Digital Medievalist.

The goal of my procrastination this time was to get my entry recognised as a “Featured article”. A “Featured article” at the Wikipedia is one considered suitable for displaying on the site’s front page. Such articles are supposed to represent the very best of the encyclopaedia, and an unofficial policy, frequently cited by reviewers, restricts them to approximately 0.1% of the total database.

Getting an article recognised as a “Feature” turns out to be a remarkably difficult process. You nominate your work for consideration, at which point it is opened for review by the community at large. And they basically tell you to take it back and make it better. Very few articles seem to sail right through. The ones I saw on their way to featured status had all gone through the process at least once before.

In my case the reviewers didn’t like my referencing style, thought the writing was aimed at too specialised an audience, and generally wanted much more background detail. After two weeks of hard work, and about 100 edits, the article is now beginning to get good rather than lukewarm to negative reviews and now seems on its way to getting recognition as a “feature”. I’m debating resubmitting next time I have something else to avoid doing1.

In addition to being surprisingly conscientious, the comments I received on my piece were also remarkably astute. Unbeknownst to the reviewers, indeed, they accurately reflected the article’s history. I the first added the entry—which is on Cædmon, the Anglo-Saxon poet and subject of my recent book from Boydell and Brewer—last year, when, trying to avoid researching an entry on fonts for the Digital Medievalist, I decided to see how the Wikipedia handled something I knew something about. The original entries on Cædmon and his Hymn were quite inaccurate, and relied on very old sources; the one on Cædmon’s Hymn also made an odd claim about hidden paganism in the poem. In the interests of procrastination, I decided to correct the entry on Cædmon’s Hymn, and replace the account of the poet’s life with an entry I had just written for a specialist print encyclopaedia, The Sources of Anglo-Saxon Literary Culture. With my print editor’s permission, I quickly revised the entry I had submitted to him, cutting out unnecessarily detailed information and adding some background material, and pasted the results into the Wikipedia. There were a couple of minor problems—I forgot to remove some books I was no longer citing from the works cited list, and some of the italics and character encoding were messed up—but on the whole the article received good comments on its discussion page, and was left alone for the most part by other users. This is generally a good sign in the Wikipedia, and in fact a criteria for recognition as a featured article.

My entry for Cædmon’s Hymn didn’t fare as well: the author of the original piece kept reversing my edits until others recommended that the piece be merged with the larger Cædmon article. I never did finish my work for the wiki entry on Fonts that I was supposed to be researching for the Digital Medievalist… though I do have an article due at the beginning of May that I’d like to avoid.

I’ve told you the story of how I came to write this paper—or rather avoid doing so—because I think it illustrates several important things about the possibilities and challenges involved in creating an information commons.

Information commons are a relatively hot topic right now. These are collaborative environments in which content is developed and published interactively—by the users of the community for which it is intended. Such communities can take various forms, but the most common are probably blog farms, RSS servers, Wikis, other types of collaborative tools such as Version Control Systems, annotation engines, and the more familiar chat rooms and email lists.

More importantly, such environments are beginning to become more popular in the scholarly world as well. A number of projects, such as STOA, the Digital Medievalist and Digital Classicist Projects, the Virtual Humanities Lab at Brown, and the Text Encoding Initiative are beginning to use tools like Wikis as a central part of their environment, or experiment with more sophisticated types of collaborative tools such as annotation and editing engines.

What my experience with the Wikipedia shows is that these commons can indeed end up with—if I say so myself—detailed and scholarly work of a relatively high standard. I don’t work for the Wikipedia after all, and I have—for whatever twisted psychological reasons—devoted a reasonable amount of time and expertise contributing a thoroughly researched and referenced entry on my subject.

Moreover, my experience also shows that such communities can be collaborative in the best sense: my article is now much better suited for its intended audience—and better written, I think—as a result of the criticism I received from the Wikipedia reviewers after I nominated it for feature status.

And, a final positive point: it shows that such communities can be self-policing. The person who thought Cædmon was really praising Pagan gods in the original entry (a very non-standard view) was eventually outvoted and reined in by a consensus among other users. And to his credit he accepted this consensus. and moved on.

But my experience also shows some of the difficulties involved in running a community of this sort:

First of all, my best and most careful work appeared only with the prospect of a reward. The reward is not great—neither my dean nor my colleagues are going to care if my article is selected as a “Feature.” But it was only once I decided to go for Feature status that I did the kind of detailed slogging that I normally do in my day-to-day research, and indeed had done in the print entry from which I revised my Wikipedia article.

Secondly, while I did contribute up-to-date scholarship to the Wikipedia, I didn’t do any research for the Wikipedia: I contributed my Cædmon article because I had something suitable lying around which I had already researched and written for a different purpose. Nobody—even the hobbyists who contribute most of the Wikipedia’s material—would put the kind of research I did into a piece written exclusively for it. If they did, it is highly doubtful that they would devote the kind of time to checking citation style and the like that print editors demand from professional scholars.

And finally, although the community is self-policing, it is not always safe to walk the streets at night: the person whose work I corrected, did, after all, come back and undo my revisions. Even though he ultimately gave in to the consensus opinion of the users—and what if the consensus had been wrong?—his inaccuracies nevertheless did replace my corrections for a significant amount of time.

I am not the first person to notice these positive and negative aspects of the commons: having used wikis on a number of projects for a couple of years, I can tell you that the problem of quality control is the second thing most academics comment on when they are introduced to wiki software, after first expressing their admiration for the concept of a user-edited environment. But because these environments are becoming more popular in a scholarly context, it is worthwhile revisiting what are in my view the two most important organisational issues facing scholarly intellectual commons:

  1. How do you get knowledgeable people to contribute their best work?
  2. How do you prevent abuse/vandalism/and/nonsense from the well-meaning but incompetent?

For the rest of this paper, I’m going to address these problems in a fairly speculative way. There are some practical steps we can take right now to find solutions to them but it is worthwhile also thinking about how they might be solved given enough time and technical expertise. Indeed in some ways, my goal is to contribute to a debate in the much the same way one contributes to the Wikipedia: throw something out there and hope that somebody can improve on it.

Although these are crucial problems for intellectual commons, they are by no unique to them. The question of how you get good quality work in and keep the bad out is also central to the operation of peer-reviewed journals or, indeed, any kind of organised communication.

These are crucial problems for an intellectual commons, however, because, in its purest state, a commons has no gatekeeper: the Wikipedia is the encyclopaedia that “_anybody_ can edit” (emphasis added). That is what makes it so exciting but also causes all the problems. Traditionally, scholarly journals and academic presses (organisations that rarely pay their authors) have addressed this problem with a combination of carrots and sticks: they encourage people to contribute by providing enough prestige to make it worth their while to submit well researched articles, and they keep the bad stuff out by getting disciplinary referees to review the submitted articles before they are printed.

A true intellectual commons lacks both a system providing rewards and preventing folly. Perhaps for this reason, most academic commons rely on some kind of gatekeeper: you need to be approved by a moderator if you want to join a mailing list; you need to submit a CV if you want to be able to annotate an online edition; you need to have your login approved by a Sysop if you want to contribute to a scholarly wiki. Even beyond this, such projects also usually engage in editorial control: spammers are cut off, trolls and flamers are banned, and wiki or annotation contributions are reviewed for quality by some central person or group.

These approaches are effective on the whole at preventing or mitigating abuse by unqualified or out-of-control people. They do, however, suffer from two main problems:

  1. They scale very badly: while a gate keeper or moderator can vet or edit contributions from a small number of people, this gets progressively more difficult as the community expands.
  2. They represent a compromise on the thing that makes commons different and exciting in the first place: the possibility for unnegotiated collaboration and exchange.

Scaling is probably not an issue for most academic projects. Digital Medievalist is a relatively big project now, for example, and it is only approaching 250 members. Numbers like this are relatively easy to control. The costs one would incur in trying to develop an automatic vetting system for a market this size would vastly outweigh any future benefit.

Other disciplines, however, have been faced by this scaling problem—and managed to find partial solutions that in my opinion do a better job of maintaining the unnegotiated quality that make successful commons what they are.

One solution commonly proposed solution is to rely on distributed moderation—or, in simple terms—allow the users to police themselves. This has the advantage of being completely scalable—the number of moderators increases with the number of users. As we saw in my experience with the Wikipedia, moreover, this system actually actually can work: many (perhaps most) errors on the Wikipedia are corrected after a while and unqualified or insincere contributors often do get reined in.

But of course my experience with the Wikipedia also shows the problem with this approach. If everybody can be a moderator, then the unqualified can be as well. They can, as a result, replace good work with bad as easily as others can replace bad work with good.

A solution to this problem is to allow moderation only by respected members of the community. This is the system at Slashdot.org, a newservice for technological news. There contributors acquire a reputation based on other’s opinions of their contributions; those with high reputation scores are then added to a pool from which moderators are drawn each week (the system is actually much more complex, but the details are not important here).

Systems such as this tend to suffer from complexity: Slashdot also has meta-moderation and nobody seems very happy with anybody else even then. Moreover, members have a tendency both to game the system in order to increase their own reputations and lower those of their “enemies”.

At Digital Medievalist, we have been thinking of a slightly different model of distributed moderation, which we describe as an apprenticeship model: in this solution, newcomers are assigned relatively limited editorial, moderation, and compositional powers. These powers then increase as one’s contributions are allowed to stand by other members of the community. Initially, one might be allowed only to correct typos; as people accept your corrections, you are allowed greater editorial powers—perhaps you can rewrite entire sections or contribute new articles. If, however, your contributions begin to be rolled back, your powers shrink accordingly: the idea is ultimately a version of the Peter Principle: you rise to the point at which you are perceived to become incompetent. The main difference is that we then try to push you back down a step to the last place in the hierarchy in which you knew what you were doing.

This method would require considerable software design, and so, currently, is outside our ability. It would have the advantage over the Slashdot method, however, both of scoring ‘reputation’ on the basis of audience’s real behaviour (reducing your ‘enemy’s’ score requires you to take the time to reverse his or her edits?) and of keeping track of reputation not by points (which encourage people to be competitive) but by permissions. A points system encourages people ask themselves how much they are worth; a permissions system encourages them to take on extra responsibility.

Moderation systems are essentially negative: they exist to prevent people from messing things up. As I noted earlier, however, commons also have the positive problem of trying to encourage good work: the most strictly refereed journal in the world, as the PMLA discovered a few years back, is no good if nobody submits articles to be vetted.

This is an area in which scholarly projects seem to have devoted less attention. While most projects with commons-type environments have explicit moderation policies, few if any I have seen have explicit reward policies. They tend to have gatekeepers but no paymasters. Perhaps as a result most also seem to be the work of a very small number of people—even in the case of organisations with large numbers of members.

Once again, solutions for this problem can be found in other disciplines. The Open Source software movement, for example, relies on high quality contributions from volunteers. Open Source groups often reward significant contributors by treating work on the project as a form of “sweat equity” that allows them special privileges: eligibility for membership on the executive or board, for example, or voting rights or even basic membership.

A second solution commonly used is to give significant contributors some kind of token that sets them apart from others. This can be as valuable as the right to administer or moderate others (Slashdot), or as minor as extra stars beside your user name in the forum (Ubuntu).

Both of these solutions can be adapted to the scholarly commons. At Digital Medievalist, we are currently putting together new bylaws that will treat contributions as a condition of membership: people who contribute to the wiki, mailing list, or journal, will be eligible to assume responsibility as board members or officers of the project (I suspect giving away extra stars for good contributions might not be as effective in the academic world as it seems to be in the Open Source one—though given how important psychologically such distinctions are, perhaps they would). A second possibility for reward—albeit one fraught with difficulties—might be to award named authorship of material on which an contributor has worked: either by naming people at the top of the article or adding them as contributors to a colophon on the project as a whole.

The two easiest solutions to this problem of reward, however, are probably those used by the Wikipedia to get me to revise my article on Cædmon rather than work on this talk: offer special status for particularly well done work, and design the project as a whole so that it is a natural outlet for work that otherwise might not be used. At the Digital Medievalist, we already run a peer-reviewed journal alongside our wiki-based commons. A project with a different focus might certify certain articles in some way: as “refereed” vs. “open forum”, perhaps, and identify the authors and major contributors. Our project, moreover, is set up to provide a forum in which users can publish material they might otherwise find hard to use in furthering their careers: solutions to technical problems in digital humanities such as the development of stylesheets and databases, that are not commonly published by the major disciplinary journals.

The intellectual commons represents a new, purely digital approach to publication and the dissemination of scholarship. It is a model that cannot be replicated in print, and it is a model that many scholars feel intuitively at least will become a major force in the future of scholarly communication. In order for it to realise its potential, however, we must first find an economic model that encourages us to contribute our best work and provides for some kind of quality control—without sacrificing the very spontaneity that defines this new means of communication.

So why should I write for your Wiki? Until we answer this question, the Wiki will not live up to its full scholarly potential.

1 Update: The entry ultimately gained feature status.

----  

Back to content

Search my site

Sections

Current teaching

Recent changes to this site

Tags

anglo-saxon studies, caedmon, citation, citation practice, citations, composition, computers, digital humanities, digital pedagogy, exercises, grammar, history, moodle, old english, pedagogy, research, student employees, students, study tips, teaching, tips, tutorials, unessay, universities, university of lethbridge

See all...

Follow me on Twitter

At the dpod blog