O is for Outcomes

New campus, American University, Cairo

I have to say at the outset that I have an almost pathological horror of testing and assessment. All my worst teaching and teacher training experiences relate directly to issues of assessment.  I don’t mean assessment of me (although negative assessments of my capacity to teach may well have resulted from my incapacity to ‘do’ assessment effectively). I mean my assessment of my students. Things can be going along just swimmingly until the day of the test, or the day when I’m required to post a grade. Then all hell breaks loose. The cozy relationship I had built up with my class or with individual students is shattered irreparably. Often this has to do with failing a student, but just as often it has to do with a student not getting the A grade they had always got in the past. Or, worse still, not getting the one percentage point that will make the difference between continued funding or having to leave the program for good.

I don’t deny that testing – like death and taxes – is unavoidable. As Johnston (2003: 77) puts it, testing is a necessary evil. It is necessary because learners and other stakeholders need feedback on progress. (Or, arguably, they have a right to feedback).  And this is what testing does: it provides feedback, in accordance with principles of validity, reliability and fairness.

But, at the same time, testing is evil. Why?  Because it assigns a value to the learner, and, since the value is almost always short of perfection, it essentially de-values the learner. Worse, testing typically involves measuring students one against the other, thereby destroying at a blow the dynamic of equality that the teacher might have judiciously nurtured up until this point.

Testing is evil because it is stressful for all concerned, and because the conditions under which testing is conducted (separated desks, no mobile phones, etc) imply a basic lack of trust in the learners.

It is evil because it pretends to be objective but in fact it is inherently subjective. Why is it subjective? Because, as Johnston (op. cit: 76-77) points out, ‘the selection of what to test, how it will be tested, and how scores are to be interpreted are all acts that require human judgment; that is, they are subjective acts’. Ultimately, it is the tester – not the test-taker – who decides what counts as knowledge, and how you count knowledge.

And, finally, it is evil because the kind of knowledge implicated in language learning is uncountable. More on that later.

For all these reasons, I avoid, as much as I can, having to talk about testing, and have refused more than one conference invitation because the theme was in some way connected to assessment.

Dr Deena Boraie, Nile TESOL 2013

At the same time, I am fascinated by – and a little envious of – those conference presenters who seem happily to embrace the topic of testing – such as the indomitable, and utterly charming, Dr Deena Boraie of the American University in Cairo, who was one of the plenary speakers at the Nile TESOL conference last week (on the wonderful new AUC campus – see pics).

Deena presented a lucid, non-technical rationale for the need for ‘assessment literacy’ on the part of teachers and other stakeholders. This included some straightforward tips on how to achieve validity, reliability and fairness in teacher-designed classroom tests. With regard to test validity, Deena’s recommendation is that tests should be judged in terms of how faithfully they reflect curriculum goals, typically encoded as learning outcomes. If the desired outcome is vocabulary knowledge, this should be reflected in the test. If it is reading ability, ditto.

While this makes perfect sense, it does rather sidestep the fact that the very notion of outcomes is not an entirely unproblematic one. For a start, and as I suggested earlier, language learning does not lend itself to easily quantifiable outcomes. Johnston again: ‘Neither language nor competence in language is naturally measurable’ (op. cit: 83).  (He might also have added that teaching is not naturally measurable either – a conundrum for those of us who have to grade teachers). He continues: ‘The fundamental immeasurability of language competence lends a further moral dimension to our work in language assessment; the decisions we are forced to make about how competence will be assessed are always subjective and thus can only be rooted in our beliefs about what is right and good, beliefs which, we must always acknowledge, could be mistaken’ (ibid. emphasis added).

That’s not the only problem with outcomes-driven testing. An obsession with pegging learning to preselected and minutely-detailed outcomes now pervades every aspect of education (as I am discovering at the moment at my own place of work). Where does this love affair with outcomes come from?

Some would argue that it comes from the world of business, from what has been dubbed the ‘marketization of education’. As Gray and Block (2012: 121) gloss it, ‘In such an educational climate, students are increasingly seen as customers seeking a service and schools and teachers are, as a consequence, seen as service providers. As this metaphorical frame has been imposed… the semantic stretching of keywords from the world of business… has become commonplace. Thus terms such as “outcomes”, “value added”, “knowledge transfer”, “the knowledge economy” and above all “accountability” have become part of the day-to-day vocabulary of education’.

In an invigorating swipe at the culture of accountability, Frank Furedi, a sociology professor in the UK, condemns outcomes-driven education as ‘a technique through which a utilitarian ethos to academic life serves to diminish what would otherwise be an open-ended experience for student and teacher alike.’ And he adds, ‘Its focus on the end product devalues the actual experience of education. When the end acquires such significance, the means become subordinated to it’.

The means become subordinated to the ends. Isn’t this, finally, the real problem of testing?

References:

Gray, J., & Block, D. (2012). ‘The marketisation of language teacher education and neoliberalism: Characteristics, consequences and future prospects,’ in Block, D., Gray, J., & Holborow, M., (eds) Neoliberalism and Applied Linguistics, London: Routledge.

Furedi, F. (2012) ‘The unhappiness principle’, The Times Higher Education Supplement,

http://www.timeshighereducation.co.uk/story.asp?storycode=421958

Johnston, B. (2003) Values in English Language Teaching, Mahwah, NJ: Lawrence Erlbaum.