Feedback They Can Use:
Understanding the Undergraduate's Frame of Reference
Robert Runté, Tom Dunn, and Barry Jonas
Starting from an analysis of the characteristics of "satisfactory" and "excellent" student writing, we argue that much of the feedback currently provided to students is actively counter-productive. Improvement in student achievement requires a change in how these students view university, their instructors, and the role of writing, rather than additional instruction in either the subject content or the mechanics of paper composition. For example, we argue that the most crucial feature of writing at the marginally "satisfactory" level is the absence of "ownership": many undergraduates believe that they "have to give the professor what s/he wants hear". It is this (mis)understanding of the university context which denies students ownership of their papers, and so ultimately undermines their potential for learning and achievement. Consequently, changes are needed in the way we structure and grade assignments to allow these students to take their own voice seriously and to develop the other metacognitions necessary to achieve excellence.
Many university instructors devote long hours to marking, often taking great pains to provide students with thoughtful, substantive feedback. Unfortunately, these comments are usually written from the instructor's perspective, not the student's. Unless instructors have a clear understanding of their students' frame of reference, they will be unable to provide feedback which is meaningful to, or useable by, the student. For many students, perhaps even the majority, our careful, time consuming critique of their work is simply so much wasted effort. Indeed, for weaker students, it may actually prove to be counter-productive.
In this paper we wish to accomplish three things: to identify the characteristics that distinguish successful from less successful student work; to relate these categories to the students' frame of reference; and to suggest some simple changes that would allow instructors to provide more student-oriented assignments and feedback. We believe that changing from instructor-centric to student-oriented feedback would significantly increase both teaching effectiveness and student success.
Understanding Differences In Student ApproachesNot all students approach a given assignment in the same way. Successful students bring with them a whole range of attitudes, prior knowledge, and composition tools that allow them to turn out a successful paper or essay examination, even when not that familiar with the actual course content. In contrast, less successful students are often defeated before they start by a deeply entrenched set of misconceptions about what is required and how to begin.
As instructors, we tend to focus our feedback on specific course content, or perhaps some aspect of the student's written communication: flaws in grammar, structure, organization, and so on.
Both groups of students receive the same type of advice from markers, but only the first group, the minority of high achievers, can actually take full advantage of marker feedback. The second group lacks the context in which to correctly interpret the feedback they receive, and even if they could, it is usually not the feedback they need. Although these students may also have weaknesses related to course content and written expression, their more fundamental need is for feedback addressed to the level of their metacognitions. To succeed, these students need to change their basic approach to the writing task, to be able re reorient themselves to each new specific assignment, to take ownership of their voice and ideas, to feel safe to take more risks, to be able to tolerate more ambiguity, and so on. Yet these are not the issues on which marker feedback is generally focused.
To understand the context in which students approach each new assignment, it may be useful to examine the common characteristics of successful and less successful student writing. A study of student written work on Alberta's school-leaving examinations in English and Social Studies turned up an unexpected result: Students producing "satisfactory" work (a "3" on Alberta's 5 point marking scale) seem to adopt the same approach regardless of subject or task. There were startling similarities in how these students approached a topic, put ideas together, and used language. Features of "excellent" work (papers receiving a 5 out of 5) were also similar from subject to subject.* Alberta teachers subsequently began to refer to the qualities of "3ness" and "5ness" that distinguished "satisfactory" and "excellent" papers respectively. Analysis of these characteristics yielded four major themes or dimensions: ownership, ambiguity, risk-taking, and specificity. These findings are summarized in Table 1,.
Although issues of confidentiality prevent us from directly quoting student papers, it may be useful to reconstruct two examples to illustrate some of the characteristics of "3ness" and "5ness". Typically, a paper exhibiting qualities of "3ness" will begin its discussion of, say MacBeth, with an opening sentence such as:
One is immediately struck in these essays not only by the meaninglessness of such verbiage, but also by the desire to please the marker. One gets the strong impression that such students have simply memorized the fact that teachers think Shakespeare was a great playwright (why else would they have included him in the curriculum, after all) and are determined to assure the marker that they think so too. In truth, these students would rather impale themselves on their protractors than voluntarily read any Shakespearean text. The lavish praise these students heap on the texts such as Macbeth have little relation to their own opinions, as becomes painfully evident when they reveal their lack of understanding of what the play is about. These are the students who believe it is possible to summarize MacBeth in a single paragraph, who adopt the safest interpretation they can grasp, and who leave the examination hall convinced they have aced the test.
Table 1: Qualities of "3ness" and "5ness"
In contrast, the "excellent" papers generally take an original, and occasionally contrary, position on what has been taught. A typical "excellent" paper might start by complaining that MacBeth is over-rated compared to Shakespeare's other tragedies, and that the standard interpretation entirely misses the significance of the symbolism of the witches. The "excellent" student takes risks, and is confident that the marker will reward such risk-taking. Reading these essays one knows that these students have internalized both the play and its meaning: they have made MacBeth their own.
For example, one year the examination asked students to discuss the theme of "alienation" in literature. One student chose the poem "Examiner" to illustrate his argument. This already indicates a willingness to take risks, because poetry is almost always more difficult for students to interpret than a prose piece. Furthermore, it was a unique and daring selection. The poem the student argued, is about the alienation felt by a teacher who is forced to proctor an examination, but who sees himself doing to his students what the gardener outside his window is doing to the grass: scything it to an even height. At one level, the paper was a straightforward discussion of a poem that met all the requirements of the question posed; but at another level, the student had built in a complete subtext, that of blowing a giant raspberry to the examiners marking the paper. Pretty sophisticated stuff for a Grade 12 student writing first draft in an hour and a half examination. The teacher-markers loved it. They gave the paper the highest mark in the province. And yet, we would not be surprised to learn the student had left the exam with a certain amount of trepidation, perhaps even thinking, "They're going to crucify me."
Although the characteristics outlined in Table 1 were derived from an analysis of school-leaving examinations, they are likely to have considerable resonance with anyone teaching at a post-secondary institution. Table 1 essentially describes the range of student writers that constitutes the universities' intake. In other words, the Diploma Examinations provide the "before picture", the starting point from which students enter the university system.
If our purpose is simply to sort students from best to worst, to certify success and failure, then our current grading practices will appear to do the job well. That portion of our intake that arrives as "excellent" students will continue to be able to analyze and refine their work in light of marker feedback, while less successful "satisfactory" students will slowly drown in a sea of incomprehensible red ink. If, on the other hand, our purpose is to allow all students to achieve to their highest potential, then our current system is inadequate and does a disservice to the "satisfactory" portion of our intake. Although these "satisfactory" students arrive with the minimum skills upon which to build a successful professional or university career, they are more likely to be ground down by well intentioned but misdirected feedback that fails to address their needs. We need, therefore, to develop a better understanding of the student's frame of reference, particularly as it applies to the potentially successful but struggling "C" student. In the following section, we suggest some of the issues which may intrude between the feedback provided by the marker and the student's ability to interpret it. In the concluding section, we will then recommend some simple changes to assignment design and marking practices to help students overcome these misconceptions and so produce the better work of which they are capable.
Student (Mis)Perceptions of Written Assignments & Feedback
A fundamental weakness of many of the students entering university at the "satisfactory" level is their lack of any sense of ownership or voice in their approach to written work. There are several aspects to this problem.
Giving the Instructor What S/He Wants to Hear (Content)
How does one convince students that they do not need to agree with the instructor to do well in a discussion course? Whenever students are challenged to take ownership for their assignments and write what they truly believe, the vast majority complain that their past experiences have shown that to do so is to take unacceptable risks. One must, they argue, agree with the instructor's opinions or pay a heavy penalty.
This nearly universal perception on the part of students does not accord with the experiences of most instructors, who generally feel they bend over backwards to accommodate other viewpoints. Similarly, there are always a few top students who will attest that that no one has ever failed them for disagreeing with their professors --- and that they routinely play devil's advocate and attack all their instructors' pet theories because they find arguing against a known position helps to focus their own thinking. Have these students been outrageously lucky in their selection of courses and instructors, or are all these other students wrong when they claim they have to "suck up" to do well? What we have seen at the high school level, what we know happens there because we have been part of the team that designs the exams, that trains and supervises the markers, and then does follow-up research, is that the students who do the best almost never spout the party line. On the other hand, there may be just enough truth in the students' fears to perpetuate this paranoia. We all have our suspicions about one or two of our more opinionated colleagues, and it is quite possible that a few bad apples could be tarring the reputation of the entire professorate. More likely, however, is that we are all a bit biased, whatever our intentions. We may think we are basing our marking entirely on the sophistication of the student's analysis, but it is obviously easier for them to be sophisticated if they base their approaches on sophisticated theorists; that is, those with whom the marker agrees. Students can adopt a contrary position and do well, but only if they address the weaknesses inherent in the out - of - favour theory, or anticipate and refute the instructors' potential counter arguments; only, in other words, if the student does more work than one who simply reiterates the positions with which the marker agrees.
We suspect, then, that what is happening is that stronger students do best when they demonstrate their independence of thought by taking the instructor on; while weaker students' papers collapse under the rigorous examination that follows their adoption of an unpopular position. Weaker writers who toe the party line get mediocre marks for lack of originality but are accepted as basically correct. Stronger students who adopt the party line do better than weak writers, but not as well as they could have done had they written from their heart. The instructors and top students are then confirmed in their belief that the system is completely fair, while the lower students learn that schooling is about compliance and conformity.
Giving the Instructor What S/He Wants to Hear (Process)
It should come as no surprise that many students believe that they have to conform to the arbitrary and idiosyncratic demands of their instructors when no one has ever explained to them that different disciplines, traditions, and paradigms may call for different approaches to the writing task. The student who has successfully internalized the post-modernist concern that writing preserve the voice of those quoted with as little interference by the author as possible, is likely to be frustrated by the business instructor who declares that s/he "stopped reading after the second page because no executive summary should be longer than a 1000 words". The business student is similarly confounded when told by the comparative literature professor that a concise two-page summary of complex material is unacceptable because it lacks sufficient embellishment and fails to allow for a multiplicity of discourses. Over generalizing the feedback received in one course to assignments in other disciplines where the advice may not apply, students naturally conclude that these contradictory directives reflect the personal prejudices of individual instructors rather than systematic differences between disciplines.
Indeed, many university teachers, isolated within a departmental organization that restricts their interactions to colleagues within their own or related disciplines, may themselves be only vaguely aware that other departments require significantly different approaches to written assignments. Secure in the knowledge that all their close colleagues stress the same attributes, many instructors do not even articulate these criteria for students ahead of time, but simply dock marks after the fact for any deviation from their discipline's implicit standards. Because grading criteria are not made explicit, and explicitly linked to the particular assumptions of the discipline that provide their rationale (or at least their history), students have no basis on which to predict these hidden criteria.
When students experience variations in the writing task as something imposed by the instructor, rather than as defined by the logical requirements of particular applications, they have no basis on which to make informed decisions for themselves. Instead of being provided a set of tools from which they could reasonably choose the most appropriate tool for the task at hand, they are left with the mistaken impression that all writing tasks are the same, and it is only the instructor that changes. It should be obvious that it is impossible for students to take ownership of their writing if they feel its characteristics are arcane and dictated by the instructor.
Giving the Instructor What S/He Asked For
Paradoxically, when instructors are clear about the disciplined-based requirements for their assignments, many students at the "3" level take the directions too literally. Instead of producing an integrated and coherent argument, they mechanically reproduce the assignment structure or scoring rubric, often even adopting the descriptors from the rubric as their topic headings. Such methodical approaches do allow limited success, because it is clear to both the student and instructor that the student has met the assignment's minimum requirements. This approach is further reinforced by reasonable success in those course components (such as labs) where essay-style integration is not required. Their unwillingness to take risks, to bring their own out-of-course knowledge to bear, or to push the envelope, however, restricts their grades to the "C" and "B" ranges. For students who religiously follow the structures set out in the assignment, it is frustrating to be told that they received only a minimal pass because they "failed to go beyond" the requirements as set out in the assignment. "Tell me what you want," they complain, "and I'll do it!"
Writing "Like an Academic"
A somewhat related problem is that in trying to enter the relatively foreign world of academic discourse, many students fixate on the tropes without understanding their substance. The most obvious and most frequently encountered example is the inflated diction of students who lack confidence in their own voice. Faced with readings that appear nearly incomprehensible, students often strive to make their own writing sound equally important. In attempting to emulate this supposed academic style, they often sacrifice what clarity of thought and language was available to them.
Similarly, some students adopt the stylistic devices of particular paradigms -- such as the jargon and rhetoric of post-modernism, Marxism, feminist critique -- without mastering the content that underlies these specialized discourses. Fending off constructive feedback with accusations of marker bias, they often fail to improve.
Writing to the Instructor
Because students know that the instructor is only person to read their paper, they often mistakenly treat their assignment as a personal communication, rather than as an essay intended for "publication". This unfortunate confusion of formats often undermines student success. Weaker students are unlikely to grasp the importance of specificity, for example, when their only audience is the professor, whose assumed expertise in the subject seems to the student to make it unnecessary, and perhaps even presumptuous, to provide supporting detail. Similarly, students occasionally document their process -- listing every book read and web site viewed in the order in which they were encountered, describing how each draft has been revised in light of new insights, and asserting how much they have learned -- without actually making a coherent argument or directly answering the question posed.
We believe that there are four attitudinal changes required of instructors to achieve these goals.
First, instructors in all disciplines need to attend more closely to the writing process. For some students, achieving a Satisfactory (3) level of performance is a great accomplishment. For others, that level of achievement is only the beginning of what is possible. In many subjects, however, instructors take the students' writing skills as a given, and feel little or no responsibility for helping to improve them. Fixated on subject content, these instructors feel that the writing process is outside their purview. Consequently, students writing at the "3" level are progressively marginalized as they advance to their second and third years with their writing difficulties unaddressed. It is generally left to teachers of rhetoric, English literature, and so on to accept the full burden of instruction in the writing process. We believe this is an error, and that all undergraduate instructors bear responsibility for all aspects of student learning, including (perhaps especially) the writing and thinking skills that are basic to mastery in any discipline. Assignment structures, scoring criteria, and marker attention all have to be directed to the writing process (as well as to the subject content) in every course in which written assignments are set.
Second, instructors need to provide students with constructive feedback that makes sense within the students' own frame of reference. Grades alone are insufficient, since these often appear arbitrary, even random, to students. We are all familiar with the student complaint that papers that were dashed off in a single evening may do as well or better than those in which the student invested inordinate time and energy. Students need extensive written feedback, explicitly linked to clear scoring criteria, to be able to make sense of their grades. For students in the "3" category, this feedback must focus on the writing process at least as much as on content weaknesses. Equally important, however, is that these comments be linked to the requirements of the particular writing task as shaped by the discipline, school or paradigm. Instructors need to recognize that these students may well be receiving contradictory advice in another class, and without a discipline-based rationale, have no context in which to interpret this conflicting feedback as other than marker bias.
Instructors also have a bad habit of making only negative comments. Taking student strengths for granted, they tend to focus their comments on those areas where they see the student needs help. But that approach reflects the instructor's perspective, not the student's. All the student sees is a sea of red ink, a litany of complaints and criticisms that undermine confidence and do not promote learning. For students to successfully integrate the feedback they receive, it needs to be balanced with at least as many comments on what they have done right as suggestions for improvement. Given the occasionally contradictory feedback from different disciplines, it is extremely useful for students to have their strengths identified so that they know what to do over again.
Furthermore, feedback is most effective when it begins with student strengths and then moves to the corresponding weaknesses. For example, it is not uncommon for a student to use three examples in building their argument. If the first example is well developed, the second less so, and the third much weaker -- perhaps even off topic, as the student desperately casts around to build their case -- one's feedback should begin by identifying what made the first example so powerful, before turning to demonstrate the lack of those same qualities in the third example. Similarly, a student who has used parallel structures to good effect in one part of their essay, but suffered breakdowns in parallelism elsewhere, will be better able to understand what they should have done when the marker starts by explaining what went right in the first instance. By building on student strengths, by making the student conscious of what they are doing right, one is able to move students from unconsciousness to consciousness and so towards the ultimate goal of automaticity.
Third, instructors need to provide students with greater opportunities to undertake
writing tasks that are meaningful within the context of the students' careers and lives.It is difficult to see how students are to take ownership for their writing when they are seldom given topic choice or a real audience to address.
Finally, instructors need to structure written assignments and scoring criteria to elicit desirable characteristics in their students' writing. They also need to provide direct instruction on the writing process as it relates to their discipline.
Dunn, Tom and Barry Jonas, "Moving Students Beyond Satisfactory Writing", Summer Institute on Student Assessment in the Classroom, Calgary, Alberta, August 16, 1994.
Runté, Robert, "Replacing the Term Paper with a Website: Some Practical Considerations", Society for Teaching and Learning, June, 1998.
Runté, Robert. "On Giving Professors What They Want to Hear", I'm Not Boring You Am I? 1:8 (Winter, 1990) FAPA 212.
Runté, Robert., Barry Jonas and Tom Dunn, "Falling Through the Hoops: Student Construction of the Demands of Academic Writing" in, A.Stubbs and J. Chapman (eds.) Rhetoric, Uncertainty, and the University as Text: How Students Construct the Academic Experience Regina: Canadian Plains Research Center, 2007.
Scraba, Elana. Patterns & Processes: Approaches to Writing by Grade 12 Students Edmonton: Student Evaluation Branch, Alberta Education. (undated)
Taylor, Gerald and Robert Runté. "Sociology, Student Directed Inquiry, and the First Practicum", Curriculum and Teaching, Vol. 6, No 1., 1991
Barry Jonas was the Exam Manager of the Social Studies 30 Diploma Examination with the Student Evaluation Branch of Alberta Education prior to his retirement.
Robert Runté is an Associate Professor in the Faculty of Education of the University of Lethbridge, and a former Test Development Specialist with the Student Evaluation Branch of Alberta Learning.
Comments to Runte@uleth.ca
**See, for example, Taylor, Gerald and Robert Runté. "Sociology, Student Directed Inquiry, and the First Practicum", Curriculum and Teaching, Vol. 6, No 1., 1991.