“Where’s your evidence?!” Given the vehemence with which some methodological matters are argued (especially on blogs), it’s surprising that this question doesn’t come up more often. Well, a good place to start looking for evidence might be The British Council Directory of UK ELT Research, compiled by Shelagh Rixon and Richard Smith, the primary aim of which is “to disseminate and share information generally in the area of UK-based ELT research”. (You can access it via the TeachingEnglish website here
).
It makes a fascinating read. I was particularly interested to find out what people have been researching at doctoral level. (Where else can you find this information, after all?) These are just some of the topics that have recently been investigated, and whose findings I’d love to get my hands on:
- The lexis and grammar of English as a Lingua Franca
- The use of interactive whiteboard technology
- ‘Nativespeakerism’ and the status of non-native teachers
- Formulaic language and SLL
- A systemic view of emergent course design
- Collaborative learning via e-mail discussion
- Group influences on individual learner’s motivation
And this one, not least because it is research about the effects of research:
Andon, N. 2008. What roles do theory and research play in language teaching? A case study on the task-based approach in language teaching.
The researcher’s aim was “to examine the ways that language teachers make use of theory and research presented to them in the professional literature and on training courses”.
As both a writer of ‘professional literature’ and a teacher educator, this goes to the heart of what I do. I’m often accused (and probably guilty) of selecting research evidence to support my own point of view, and ignoring that which doesn’t; or, worse, of not having any evidence at all. This is particularly the case with the Dogme ELT philosophy: it’s not enough to wheel out a supportive bibliography in order to situate Dogme on a firm theoretical base (as I did on Jeremy Harmer’s blog recently). Nor will anecdotal evidence do: the Dogme discussion list is strewn with feel-good accounts of ‘successful’ materials-light, talk-driven classes. But people want concrete proof. They want research evidence.
Fair enough. But what kind of evidence would that be, and what’s the guarantee, anyway, that this evidence would satisfy the sceptics?
Let’s take Dogme: how could you provide convincing evidence that it works? Here are some possible lines of attack:
1. Measure the outcomes of teaching two matched groups, one taught with coursebooks, one taught without. Problems: too many variables (teacher, students, context factors…); what outcomes do you meaure (fluency? accuracy?) and how do you ensure your assessment criteria don’t automatically favour one approach over another? Also, it would probably need to be done over an extended period to produce significant findings.
2. Record and transcribe a sequences of ‘Dogme-style’ lessons, and track ’emergent language’, i.e. language that learners have seemingly appropriated and then re-used subsequently, thereby showing that learning can take place without a pre-selected syllabus and solely through interaction. Problems: an enormous amount of work (all that transcription); you would also need to pre-test – but what would you be pre-testing for if the learnt language is not pre-selected? Also, without a control group, there’s no way of knowing if that same language would also have emerged in a more orthodox setting.
3. Ethnographic case study of a Dogme class over an extended period, using observations, interviews, questionnaires, etc, to gather a ‘thick description’ from the point of view of the participants. Problems: nothing to compare it with; too context specific, hence ungeneralisable; ‘Hawthorne effect’, i.e. subjects out-perform when they know they are being experimented on; attitudinal questionnaires are unreliable – subjects say what they think you want to hear.
4. Fine-grained, micro-analysis of classroom interactions in a Dogme class compared to a ‘traditional’ class, to demonstrate, for example, a greater quantity of and/or better quality of communicative, scaffolded, authentic, creative, etc language use in the former. Problems: again, assuming you could control the variables, the specificity of the findings is unlikely to satisfy the sceptics; also, because the findings are evaluated through the lens of a specific theoretical model – e.g. sociocultural learning theory – your conclusions depend on this theory being generally accepted – which it isn’t.
Have I missed anything out?
In the end, though, there’s probably nothing you can do to convince the doubters (let alone the cynics). Which makes one wonder: why do research at all? One way of answering this question might be to re-assess what research is capable of achieving. Nunan (1992) distinguishes between two alternative conceptions of research: “The first view is that external truths exist ‘out there’ somewhere. According to this view, the function of research is to uncover these truths. The second view is that truth is a negotiable commodity contingent upon the historical context within which phenomena are observed and interpreted” (p. xi-xii). Researchers in the second tradition are interested less in proving a theory than in deepening their understanding of their own situated practices. This understanding may, in turn, influence the way these practices evolve.
But isn’t this a cop-out? Is there no way my research can be generalised to your context?
Reference:
Nunan, D. (1992). Research Methods in Language Learning. Cambridge: Cambridge University Press.
An A-Z of ELT 
Hi Scott,
I know it’s a bit heretical to admit it, but no, I for one generally don’t think “your” research can ever really be generalizable to “my” context, for exactly the reasons you point out in your number 1 above. Most of all, I think there are too many variables to control for, especially if you are looking at something so broad as the efficacy of an entire approach. One or two classes would surely not convince me; you’d need hundreds on either side, and measure them over time as well. Even then, I might be liable to come away thinking that teachers who *self-identify* as dogme tend to be more or less effective than those who don’t, rather than thinking that it’s due to the approach.
That doesn’t mean that such research would be worthless, of course. It is certainly useful in that second tradition you point out via Nunan in the end. Perhaps, as a poem can hint at general truths without being literally “true,” the research of others can give us inspiration on how to better explore our own classroom practice. Asking for more than that would be like asking for a sonnet to definitively prove that being in love is more worthwhile than not. Even thousands of poems over time can’t prove that–but reading them surely gives us some very important insights.
Marcos
Thanks Marcos – I love your analogy with poetry (which also connects nicely with a previous topic on this site!) and it reminded me that my favorite adjective, when describing any research that interests me, is ‘suggestive’: ‘The study by X about Y is very suggestive…” etc.
The same adjective also applies to a lot of love poetry, although perhaps with slightly different connotations!
The question arises: who is the research for? For, as Nunan suggests, the researcher her/himself or for the benefit of the wider world of ELT? And then who can access and read it?
Since starting my MA, I’ve read many articles and accounts of research that have made me reflect on my own teaching practice. In some cases I have made changes to what I do in class, in others I have judged it an intresting but not applicable to my context. However, if I wasn’t engaged in academic study, I would never have heard of these papers and would have got new ideas from ‘anecdotal’ blogs instead…
I think there are no solid truths ‘out there’ to be found. Everyone has their own set of beliefs that they bring to teaching and challenging those is difficult. As you say limitations of the study will always be cited as will factors of context. However, I believe research is important as a way of highlighting ideas, theories and factors that influence language learning in an organised, vetted way. Besides, how would I write my next assignment without it? 😉
David’s question is a good one – who is the research for and to what end, given that in the end, you will probably not convince anyone else?
I took on a large project last year, to learn from dogme/dogme 2.0 and pre-tested my students using an external, not ELT-publisher affiliated, summative examiner (www.mondiale.de).
I taught the students in both classes with zero/little materials and in a computer training room with the web2.0. Thus, my pre and post-testing of them was highly useful.
I have since written up a case study on the project, which will probably only be read by a handful of LT academics (if I am lucky) but while I was fighting with my own computer’s problems and kicking myself for agreeing to translate this research into something that would be available for others to read, I found out that, most importantly, it solidified my own direction as a dogme practicioner.
The writing – the reflection – taught me a great deal of what I was actually attempting to do. Through seeing where my students’ progress occured and where it did not and what they themselves thought about where they were improving most, reviewing their feedback forms and then comparing this against the results form Mondiale for the solid “data” to refer to I was able to get a more accurate picture of dogme’s affect – see what wasn’t working also – be it dogme or dogme 2.0 and make adjustments in my teaching practice.
So.. I think, um, the point I am trying to make is that while it is all well and good for research to be something that can be published eventually and possibly read by someone else, the most important point with regard to putting your beliefs and practice under the microscope is actually to challenge yourself and to stretch beyond possibly fossilized beliefs; to learn from your own mistakes, to find the gaps – things which you can only see once when you have to compile up months of research into a ‘readable document’ and of course, to have the chance to kick yourself for not having set up a system to record x or y… all in all, in the end, to become a better teacher and teacher-trainer.
Hope that makes sense,
K
I like to think so, to, Karenne – and you put it very nicely. At the same time, wouldn’t it be nice if this ‘microscopic’ work that you refer to had a certain generalisability to other contexts, so that you could answer (e.g. to the dogme sceptics): “It DOES work in context X, so there’s every reason that – given appropriate adjustments – it should work in context Y”. (This leaves wide open, of course, the question as to what the ‘appropriate adjustments’ might be).
I’m going to comment in a slightly different direction. I’ve always been annoyed at conferences with people presenting on research outside my context. If I’m at a conference in Turkey, I’m not really interested in “Reflections on Tech with Japanese Students.” This research is invariably from one, perhaps two classes at most and so far outside my context I don’t really see the point.
Research is important and, yes, I do think something can be gleaned from it, but when choosing such a specific topic for a conference, present it at a conference within that context IMO.
I know how you feel, Nick – although I think the problem is less about people presenting their research than their failure (often) to suggest how their findings might map on to the local context, or, at least, to invite their audience to make these connections. This, in turn, is often an effect of the limited time available (or the poor use of the limited time) so that a one-hour presentation turns into 55 minutes of data presentation and only 5 minutes for discussion.
One of the things I like about research is the fact that intuition doesn’t always cut it. I taught EFL for something like six years before doing an MA, and I was amazed to find (on my MA) that a lot of what I thought was intuitively correct (such as teaching vocabulary in lexical sets (thank you course books :D)) was, according to the relevant research, a burden to learners and not doing them any favours at all. (Of course, upon presenting this research as part of a vocabulary workshop I did at a private language school I worked for, I was told by one of my superiors that he wholeheartedly disagreed with this (based on his intuition, naturally). However, I still think that certain research, such as this, can be considered generalizable.) In essence, I think that (good) research is helpful when it comes to cutting through long-held personal beliefs/assumptions about learning.
Still, having said that, research can be incredibly slippery – so, like anything, I think we have to search for the good stuff, and there is a lot of good stuff out there (amongst the questionable stuff). (Incidentally, I see this as being much like the rest of life – not all recordings of Mozart’s Requiem are created equal, once you know what you are looking for. However, knowing what makes a great recording takes a trained ear, and I suppose it’s the same with research. In my experience, if I hadn’t embarked upon further study I’d have real difficulty in understanding a standard research article. And to top it off, I *still* have real trouble understanding the results of any sort of ANOVA test or the like…)
Finally, I agree that it’s a real shame that so much academic research is unavailable (and often inaccessible) to those practising teachers and educators who might actually be interested in reading it.
Richard, it’s a real shame indeed that academic research is inaccessible. I now have the privilege to be enrolled in a university and have large access to academic journals and resources, and that means also that I’m spending a lot of money.
In the past three years I’ve been thirsty for this kind of support to back up my professional practice but couldn’t have it, no money.
A vivid example follows:
Mr. Joe Teacher wants to convince Mr. DoS that it’s a great idea to set up blogs for the English learners, Mr Joe Teacher uses the argument that it will increase learner autonomy.
Mr DoS thinks otherwise, he says he tried once and didn’t work out, but if Mr. Joe could support his idea with ‘evidence’ he might give it another try.
Mr Joe goes to Oxford ELT Journal online in quest of research support and guess what… There in the current edition he finds Augmenting learner autonomy through blogging ELT J (2010) 64(4): 376-384
And guess what?
It reads below: You may access this article for 1 day for US$25.00.
What happens next?
I’m working on a Napster ELT btw, if anyone wants to join me let me know : )
Hi Scott,
I had written a comprehensive account of how I believe research in our field would benefit from a Complexity Theory approach. But I lost it, damn stupid browser and damn stupid writer who should’ve used a notepad to write the comment down and then type it.
anyway… I can’t do it again, so brief comments follow below:
– From a complexity theory perspective many of the things you classified as ‘problems’ are in fact the solution. E.g. too many variables (in #1). Is it possible to analyze a teacher’s behavior and disregard her students motivation?
Taking into account the notion of sensitivity to initial conditions (the butterfly effect), the problem of replicability (in #2) is invalid, in fact, unnecessary)
Other problems you mention which are not exactly problems in complex systems studies, “nothing to compare it with; too context specific, hence ungeneralisable”
– I know you know it, but other readers interested in research can get fresh insights into it on the final chapter of Complex Systems and Applied Linguistics (Larsen-Freeman & Cameron). I have just read it again after your post and it is a must read in my opinion.
– Just an interesting final remark. Coincidentally (or not), on Tuesday I got myself a print copy of this directory you mentioned and also asked Luke Meddings in person where I can find further research to support (or not) Dogme. I’m preparing my MEd research proposal and Dogme seems to be the closest I can get in terms of complexity principles applied in ELT, an innevitable springboard then.
Thanks
Willy
Thanks, Willy, for your posts – and I’m really sorry we lost your more detailed insight into how research could benefit from complexity theory.
Meanwhile, in answer to your question about further support of a Dogme methodology, a lot of the research that is being done within the task-based framework would seem to mesh quite neatly with Dogme principles, i.e. that acquisition can occur when there is a focus on form in the context of meaning-driven communication. In June this year, in Barcelona, I saw John Norris (of the Univiersity of Hawaii) give a very interesting summary of current research in this area, and that impelled me to order the collection of articles he co-edited in 2009: Task-Based Language Teaching: A reader. (You can find out more about it here:
https://www.benjamins.com/cgi-bin/t_bookview.cgi?bookid=TBLT%201
Also, studies in the field of motivation and learner autonomy offer some very suggestive evidence about, for example, learner agency. In her plenary at Harrogate this year (IATEFL 2010) Ema Ushioda mentioned a study by Leinhardt Legenhausen (1999), in which he compared Danish and German secondary school students, one group following a textbook-based curriculum, and the other learning in a more Dogme style. In freer production activities, the ‘coursebook-nourished’ students showed a lack of creativity and originality in their talk, compared to the more expressive, more natural ‘dogme-style’ learners. (I haven’t been able to get hold of the original study, but the reference is: Legenhausen, L. 1999. Autonomous and traditional learners compared: The impact of classroom culture on attitudes and classroom behaviour. In Edelhoff, C. and Weskamp, R. (eds.) Autonomes Fremdsprachenlernern. Hueber).
Also, a lot of the research that’s coming out under the sociocultural banner, e.g. by Merrill Swain etc, would seem to support a Dogme agenda – see Lantolf, J., and Thorne, S. (eds.) 2006. Sociocultural Theory and the Genesis of Second Language Development (OUP) for an overview. Rod Ellis provided a good summary of research in this field in Task-based Language Learning and Teaching (2003, OUP), while Leo van Lier’s The Ecology and Semiotics of Language Learning: A Sociocultural Perspective (Kluwer Academic, 2004) is essential reading.
And, finally, as you rightly mention, the relatively new field of complexity theory and emergentism has not only offered a new ‘metaphor’ for language learning (i.e. emergence as opposed to acquisition) but is fuelling some really interesting research studies – such as the collections edited by Nick Ellis, among others, in recent special issues of Studies in SLA, and the Modern Language Journal.
Thanks for the rich reference Scott!
I just found the issue of The Modern Language Journal you mention, the complete reference is Summer 2008, Volume 92, Issue 2 Pages 165–347
Regarding research, Cameron and Larsen-Freeman write Research Methodology on Language Development from a Complex Systems Perspective.unlocked
I couldn’t find anything on Studies in SLA, I only looked as early as 2006 though. The search engine is poorly designed and I couldn’t afford to spend over one hour browsing their archives.
Lastly, I found some papers related to dynamic systems (by Verspoor; Ellis; Lantolf; …) on the journal Bilingualism: Language and Cognition volume 10, issue 01.
Hi Willy, the SSLA special issues I was referring to are on on frequency effects in SLA (2002, Issue 24, and one on explicit and implict learning (2005, Issue 27), both edited or co-edited by Nick Ellis. For an early article by N. Ellis on emergentism, look for ‘Emergentism, connectionism and language learning’, in Language Learning, 1998, 48: 631-64. It’s a very good read. (If you like that sort of thing!) There is also a special issue of Language Learning that he and Diane Larsen-Freeman co-edited which is available as a published book, on dynamic systems etc: Volume 59, Supplement 1 (2009).
Hi Scott and everyone,
During one of my hard day’s nights through academia, a tutor of mine suggested that sharing research between contexts was like his snapping a photo of a local mountain range (substitute geographical feature as appropriate to your locality) and sending it to me. His photo of, say, the Japanese Alps, then becomes a reference and source of (visual) information I can compare and contrast to the Rockies, for example. Maybe I find my tutors photo relevant and interesting, maybe I don’t. Quality (visual detail and accuracy) are significant as are our mutual understanding of how to create the images and interpret them. Maybe I’ll learn a thing or two about composition, Japanese landscapes, or something else entirely.
With research, it’s popularization that often determines how the masses will come to understand what’s been suggested, and much can get lost in ‘translation’ when we turn the discourse of one community into discourse (in this case, popular text) for another. A prime example might be the discourse about climate change as it is popularized for National Geographic magazine and documentary films like Al Gore’s ‘An Inconvenient Truth’ for mass audiences.
Finally, I think it’s important to remember that there is a difference between fact and truth. Quite suggestive that observation. 🙂
Yes, that’s a neat analogy, Rob – i.e. snapshot. Also, I think you’re right in suggesting that research gets discovered and popularized (uncritically) because it happens to chime with whatever is trending. This happens constantly in medicine, for example: because one small study shows that red wine contains anti-oxidants, wine-lovers (like me) immediately trumpet the news far and wide! Likewise, a study that shows that – for example – learners remember better the topics that their co-learners raise in class than those that are in the book etc (as in Slimani 1992) gets a lot of mentions by people like me because this supports a more ‘dogmetic’ view of learning – even if the study was relatively small, and very local.
Hurray for red wine and Dogme! I enjoy both.
Enough of those relatively small, local studies (or snapshots), and could we have a generalizable portrait of Dogme to be applied to other local contexts?
Fascinating, as usual. I have been delaying my MA for one reason or another since I started it back in 2006. This year, I am determined to get it done. I found it difficult to identify an area for study as I labour under the illusion that teaching a foreign language is like being the tailor of the emperpr’s new clothes. There’s not an awful lot to do other than to be able to communicate well, be able to answer questions well (enough) and…errr…that’s pretty much it. Quite how I was going to research this, I had no idea.
I settled on an examination of whether the perspectives of teachers, managers and insitutions regarding lesson observations were shared. I read widely – not an awful lot about it, but plenty about how teachers probably feel about observations. Very little about how managers feel. Next to nothing about how institutions feel. There was a gap in the research-I had created my niche, but all I wanted to do was to lie down in it and have the stone rolled over the entrance.
Dogme is probably the only area that I think I could find the motivation to research whilst sat on a train at 0730 in the morning and 1930 in the evening. I approached the mountain upon which our guru has perched for ten thousand years and began the ascent. And Scott pointed me in the direction of David Hall’s facinating work with Talkbase in Thailand. I’m still doing the reading at the moment, but can already envisage a research project where students are tested upon arrival; taught for a ten week course in the style of Talkbase and are assessed at the end. People will be able to draw what they will from the results: but assuming that students show a degree of progression, these findings could be added to Hall and Kenny’s findings from Thailand; to Kenny and Lazsewenke’s findings in Vientiane and to Thein’s experiences from Thailand. In which case, research serves in much the same way as taking photograph after photograph after photograph of a mountain range. The photos can all be placed together to give us a panoramic image – and the result is often quite breathtaking.
I’m interested, Scott, in the way that you situate dogme within a range of other fields: TBL, motivation, autonomy, complexity theory, emergentism, sociocultural theory. Does this add weight to my theory that dogme is now no more than a label? Is there room on the shelves of academia for a dissertation that researches what dogme actually is or isn’t? If so, I could do that one!
Willy – never a truer word. One would mind only marginally less if the people charging £25 for a day’s access weren’t the world’s most successful academic journals publisher or if some of that money actually found its way to the author. As it is, OUP manage to uphold the academic characteristic of pimping knowledge out.
So, rather than a napster, why not create something more akin to the fanzine? An online depository of journal articles, refereed by some of the more distinguished names (to buy credibility), which aimed to uphold standards of research etc, but which would be distinguishable from all the rest because it was free. “The Free Journal” would be a catchy title.
Thanks Diarmuid. I’ll reply in more detail shortly. Meanwhile, for those intrigued by D’s mention of David Hall’s ‘Talkbase’ project, you can read more about it in the Dogme discussion list here.
I used to be an avid reader of formal research. Whether I reached a sort of “critical mass” (or critical mess?) and felt I had enough to inform me about certain basic principles, or else found what I was reading too expensive to access, too limited, too dry, too boring, too oriented around justifying someone’s tenure, too fragile in terms of reliability and/or validity, or too potentially irrelevant on account of drags in time, distance and context – I’m not sure!
I do know that I stopped looking for proof or “truths” a long time ago.
What I have become extremely interested in is blogging as a medium for doing, delivering and discussing research. There’s lots to like about it:
(a) It’s informal and focusses on being descriptive rather than trying to prove anything
(b) It’s (mostly) carried out by actual practicing teachers, concerned principally with day to day teaching challenges that more teachers elsewhere are likely to find relevant
(c) It’s done and delivered for the love and interest of it – not to earn pieces of MA or tenure paper
(d) It’s egalitarian, in the sense that a newbie teacher’s experiences and musings are just as potentially valid and interesting as the veteran with an MA degree
(e) It’s accessible to everybody, at no charge
(f) It can be challenged and discussed, wrestled and tossed about (through comment threads, hence far more interactive and “living” compared to what we find in print journals)
(g) It’s fast and it’s now
Whether some would like to admit it or not, there are already two different major streams of research going on in ELT. I think there is room for both, but for me personally, I’m much more interested in what teachers are talking about on their blogs. I also think this stream has the greater potential for growth and actual impact on what is happening in ELT around the globe, at classroom level.
Thanks for making that point, Jason. Certainly, blogging – and online discussion groups, such as the Dogme one – offer practising teachers unprecedented access to a global ‘discourse community’ of like-minded professionals, and have thereby increased the potential for in-service teacher development exponentially. Whether this should be seen as an alternative to doing (and reading about) research, I’m not so sure: it seems that the two activities are mutually supportive. An idea gets kicked around in a discussion group (such as using Second Life as a medium for language instruction, to use a recent example) and then someone picks it up and makes it the focus of a research study, the results of which are eventually fed back into the ‘long conversation’. Or they go off and sift through the available literature, synthesising the results of other studies. Either way, and even if the research is itself inconclusive, doesn’t the extra effort that went into doing it, lend it a little more authority than the purely anecdotal?
Very interesting discussion, which I shall go away & mull over… a quick aside for now though, which some people may find useful:
If you live in the UK and want to read research papers without paying for them, you should be able to access them via your local library. If your local council doesn’t subscribe to the journal you want to read, then the nearest university library almost certainly will.
Sue
Scott, I always wonder: Is there any research indicating the feasibility of conventional language training? I mean, provided that there are criteria to pin down what it is, and if somebody can be found who still claims to work with it (chances increase if GT is to be rehabilitated): Is there any hard evidence that it can be effective, in a pedagogical way? Has it ever been tested, and if yes, relative to which frame of reference?
Hi Klaus. Good question – and the person who professes to have evidence that a ‘traditional’, explicit grammar-driven approach works is Ron Sheen. He has been a fairly relentless critic of task-based learning over the years, and claims that counter-evidence is routinely ignored. See for example: ‘Tangled up in form: Critical comments on ‘Teachers’ stated beliefs about Incidental Focus on Form and their Classroom Practices’ by Basturkmen, Loewen, and Ellis’ (co-written with Robert O’Neill, in Applied Linguistics (June 2005) 26 (2): 268-274, in the abstract of which, incidentally, the authors write:
An earlier article of Sheen’s is actually relevant to the previous thread on translation: The advantage of exploiting contrastive analysis in teaching and learning a foreign language. 1996. International Review of Applied Linguistics in Language Teaching. Volume 34, Issue 3, Pages 183–198,
Thanks Scott! Now this (later) article you mentioned would cost me 25 bucks, and I don’t believe it would help me. According to the abstract, Ron Sheen complains about other people who, in his view, don’t produce enough evidence, but where does he produce his own?
I have my personal, highly subjective and un-scientific perspective. In Germany, and I guess more so in many other countries around the world, GT is still state of the art, no matter how various courses are labelled. And I see the evidence all the time, since almost all my clients have enjoyed that treatment over an extended period. They ALL (I mean that) have learned very little in a very long time. With us, with PDL, they take big leaps in a short time.
I know that already, I don’t have to prove anything, certainly not to the Ron Sheens. If I want to further improve what I’m doing, I need a different kind of research. A kind that’s not empirical, that’s based on thinking and doing and observing and reflecting and doing again. The kind that Pestalozzi did, and Freinet, Montessori, and all the others. The same should apply to Dogme, I guess.
The question remains: Where’s the research, empirical or other, that supports the notion that traditional language teaching is a good idea? I for one have reason to believe that it is not!
“Ron Sheen complains about other people who, in his view, don’t produce enough evidence, but where does he produce his own?”
This was the criticism levelled at him by Michael Long in a fiery TESOL Quarterly article in 1994 in response to a previously published critique by Sheen on task-based learning. Sheen cited some fairly obscure Methods Comparison Research (MCR) studies, one of which actually replaced the teacher with a tape recorder (as a way of controlling that variable) and found that (having removed the teacher) explicit grammar teaching was more effective than implicit teaching, presumably represented by some kind of drill-and-repeat methodology (since this was in 1970). Sheen’s allegiance to MCR is almost endearing, given how discredited methods comparison research is nowadays. For a start, the concept of ‘method’ is so shaky as to be virtually meaningless, and, anyway, the number of variables you would have to control in order to compare Method A with Method B makes any comparison highly speculative at best. This is why I’ll never be able to prove that Dogme is any better than grammar translation, task-based learning, or being hit repeatedly on the head by English Grammar in Use. And yet at my back I always hear ‘Where’s your evidence?” hurrying near.
Hi Scott and everybody,
To be honest, I’m surprised no specific research has been carried out on Dogme yet.
I agree that one of the problems with an effect-of-instruction study of this type would be how to measure gains, especially when there is no pre-defined syllabus. The Norris & Ortega meta-analysis of instructed SLA found a considerable research bias towards studies of explicit instruction, which is certainly easier to test (grammaticality judgement etc.)
As you know, they concluded that instruction does have an effect (Yay, good news for us) but there was little difference between implicit and explicit instruction, not such good news if you are proponents of implicit instruction, FonF, TBLT etc. as they are (although possibly explained by the bias)
I mention this study because it seems to chime with the analogy that David drew earlier of the snapshots. Take enough of them and you can build up a convincing picture of the terrain!
Within the case study you suggested above, the students could keep diaries to log the language they believed they had learnt from each dogme class, which would be then be triangulated with the classroom observations etc. It would be interesting to compare the learners’ and teacher’s assessment of the focus of the lesson.
This record of the language the learners believed they had ‘acquired’ from the course could then be used as a post-hoc ‘syllabus’ and examples of pre- and post-course written work could be analysed for these specific items.
Thanks, Jessica, for your (as ever) thoughtful comment. Your suggestion of using learner diaries, and comparing learners’ and teachers’ assessments of the focus of the lesson, is an excellent one. And it reminds me that I should have mentioned earlier (because it tends to lend some support to a dogme approach) the (relatively small) study by Slimani (1989) where, through a combination of observation and interviews, she found that learner “uptake” — i.e. what the learners found salient in the lesson — was influenced more by the topics that other learners raised than by the topics raised by the teacher or the coursebook. This is the kind of study that could be fairly easily replicated, and possibly enriched by some tracking of vocabulary occurrences and their later recall, for example.
I have wanted to post here for some time and will skip the less useful “What I have always thought about…” to give one reference and short comment and ask one question.
The reference:
Rod Ellis recommended this to me indirectly through a mutual friend after I had mildly shocked the YLTSIG list over 10 years ago with a fielded discussion: “Down with grammar!”
I approached this impressive article naively expecting a list of thing I should do in the classroom that had been scientifically proved to be effective. (It had been recommended to my through a mutual friend by Rod Ellis in reaction to a fielded discussion I did on the YLTSIG list entitled: “Down with grammar!”)
Here is a quote from the carefully worded findings:
Uhh? So what do we do to be researched-proven correct?
And the question (in the style of the month):
Which 10-15 research findings significantly influenced the way you teach?
Dennis Newson
Thanks for posting that, Dennis. The Norris and Ortega study (in which hundreds of previous studies were crunched and minced) is a landmark one, endlessly cited by proponents of traditional instruction, of task-based instruction, and just about everything else (except total immersion), since it really does seem to vindicate the benefits of some explicit teaching interventions in language learning. But should we be surprised? ‘Explicit teaching interventions’ is what most attentive teachers tend to do, one way or the other. But this needn’t constitute up-front grammar teaching of the type: Now I’m going to tell you everything I know about the present perfect. In fact, probably just the opposite: the case for focusing on form “integrated in meaning” (as endorsed by the Norris and Ortega study) is exactly what the dogme-tists have been arguing for all along, and constitutes what has also been called ‘teaching at the point of need’ or ‘teaching on demand’ – see the more recent thread on this blog.
Forgive me for posting two messages one after the other, but having read through all the messages before posting my last longer one, I had one further point which did not really fit in.
No one has yet mentioned so-called action research. Isn’t that similar in essence to what Jason has to say about blogs? My understanding of action research is that:
1. It is done by a practising teacher about the lessons (s)he is teaching and therefore done in the teacher’s classroom with the teacher’s learners.
2. The so-called research is focussed on investigating and reporting on what goes on in the classroom and, on this basis, coming up some appropriate future, different (classroom) action.
It has always seemed to me (oh dear, I have used the phrase after all) that academic research is normally written for, writing to the accepted rules and norms of academic study whereas what most teachers want is something written with the classroom in mind and coming out of practice and personal experience rather than being theory- or hypothesis-driven. Action research, as I understand it, is anecdotal and not rigorous, precisely repeatable, capable of validation etc. but, like blogging, does seem potentially to offer more succour to the teacher. Would you agree?
Action research doesn’t have the same sort of rigorousness as a full on academic study – but then neither does it have the same demands with regard to neutrality and empiricism. This does however make it perfect for studying and finding out about your own personal practice and development.
The value of action research stems from it being carried out by teachers for teachers, and most importantly, what the researcher can learn from it. The main purpose of action research isn’t to make broad generalisations about learning, but to find out something about your own practice. It begins with a reflecton on practical experience, but can also involve a lot of theoretical knowledge. You identify an issue in your teaching, (say, how to use more authentic listening in your class), find out more about the subject by reading other research, finding out about the theories and the models, then using this to put into place some changes in your classroom practice, which are then evaluated in some way (student feedback, self-evaluations, peer observaton, etc.), the evaluations then reflected upon and the changes either absorbed into your general classroom practice, or redesigned, or rejected based on your findings.
There has been a lot of work on action research as a research model (Jean McNiff & Jack Whitehead in education generally, Ann Burns in ELT), often looking at ways of improving reliability and validity. It is perhaps better to consider it as a kind of self-directed case study, rather than using the measures of academia, and drawing one’s conclusions from there. Anyone interested in alternatives & arguments about the value of case study research should look up Michael Bassey, especially with regard to what he calls “truthfulness” over reliability & validity, and the use of this type of research to make “fuzzy generalisations”, much of which would apply to action research.
All research, action or “academic”, needs to be viewed through careful eyes – even if I read that “activity x led to a 2.7% improvement in TOEFL scores” based on a fairly “rigorous” study of 2000 Japanese & Korean university students I would still look at the activity and think carefully about whether or not it was applicable to my context before using the activity.
Action research, if read with full critical awareness, can be as fulfilling and useful as any other, arguably more so, as it is carried out by teachers in the midst of their context. Reading it is not unlike hearing about an idea from a colleague which you can then try out for yourself, and if it works for you, great, and if it doesn’t, well, what have you lost? You might not want to base a country’s educational policy on it, and question a government who did, but it is still useful and informative for both the researcher and the reader.
Hi Scott,
I’ve just come across this intersting article by Miyazoe & Anderson in System 38 (2010) regarding ‘collaborative learning fostered by scaffolding’ in online writing activities on blogs, wikis & forums.
I’m sure I’m not the first to suggest that this might be example of ‘blogme’ (sorry, hard to resist) 🙂
Hi Scott, thanks 4 this interesting article.Actually i am interested in Dogme ELT so much and i want to do a research investigating the reasons behind its abscence in my country: Algeria where i study and teach English at the same time. I just started writing my research proposal, would you like to help me please. I would be so grateful.
hi Scott
I am Amina . i am the student of m.phil English and going to conduct resaerch but in Pakistan there is lack of reources ,kindly guide me in selecting topic related to English Langugae Teaching.