Things have a habit of coming along when you are thinking about something.

John Hatties latest book has caused a bit of a stir in education circles recently and I must admit I have struggled with some of the statements that have been attributed to him based on his research.  Then people like Arti are quoting him and appear (at least) to have a lot of time for his ideas.  Pam has her crap detector turned to max and is good at calling it as she sees it …. so maybe I am wrong?

Then this from Bruce Hammonds:

Somehow, just because Hattie has amalgamated every piece of ‘school effectiveness’ research available ( mainly it seems from the USA) his findings, it seems, ought to be taken for read. The opposite ought to be the case – we need to be very wary of such so called ‘meta research.’. More worrying however is that the approaches he is peddling is pushing into the background the home grown innovative creative learning centred philosophy that was once an important element in many classrooms. Overseas experts aways seem to know best – or those that return with their carpet bag full of snake oil.

Smythe,after reading Hattie’s book ‘Visible Learning’, writes that Hattie’s ‘feedback’ is really attached to a direct instruction process .It is more concerned with testable transmission of teacher devised content to the students and as such is antithetical to individuality and creativity. The book, according to Smythe, is ‘skewed to a certain style of teaching and learning ( learning set up for measurement) and towards appealing to conservative influences’. Enter, from the right, the School Review Office to collect the evidence, and the Minister’s National Standards to narrow teaching.

hmmmm …. had not thought of that! maybe that’s what makes me feel uncomfortable?  I have blogged before about my belief that we are not able to isolate causal factors and make stark claims about their effectiveness.  Class size, of itself, may make no particular difference (teacher quality being more important) but the things that a smaller class size enable certainly do make a difference and I would argue can allow good teaching to be even better and all the things like quality interactions in the teachable moments to happen.  And according to this we already have smaller than OECD average pupil-adult ratios.

Bruce and Kelvin have a good point about the inherent assumptions and learning and teaching model implicit in this book.  It does smack of a transmission model.  A technocratic approach to learning and the craft of teaching that Bill Gates agrees with.  Not sure I do!  everyone should take the time to read these two articles!

Then this from Graeme Wegner:

note where NZ is!  What is wrong with being 4th overall?  The Olympic Committee would all get Knighthoods if they could achieve this.  I’ll keep checking my inbox and the mail awaiting my letter of congratulations from our Education Minister (or is that a Tui advert I see?)

Then I stumbled upon this:

How can school performance be measured accurately to improve learning outcomes? “Measuring Improvements in Learning Outcomes” proposes a value-added model of measuring which provides a more fair, precise and quantitative tool for assessing students’ progress. Unlike some league tables which rely on raw test scores, value-added modelling measures what students have learnt while in school by monitoring their performance at two or more points in time. It also overcomes many of the problems plaguing other models which can be biased against schools with socio-economically disadvantaged students. The report sets out three broad objectives for using value-added modelling:

*school improvement initiatives, to better develop specific programmes.
*school accountability, to enable fairer and more accurate evaluations of performance to ensure resources are being used efficiently.
*school choice, by providing parents and families with information on the performance of different schools.

The cost of education in OECD countries rose by an average of 39 per cent between 1995 and 2004. If outcomes are to improve, accurate measures of performance are essential. This OECD report emphasises the benefits of using value-added modelling and, importantly for policymakers, discusses a number of implementation strategies available to governments.

Would love to be able to look at this publication in more detail!  But not at $50us.  [update – available HERE online] The model we have at school for our assessment tracking and reporting has a strong ‘value added’ focus.  Would be good to see what the thinking is.

So …. great to be eclectic with the places that inform my thinking and delve deeper than the surface features of the information we are given.

14 Responses to “Looking behind the numbers”
  1. I agree on this.

    Hattie has been harping on about teacher feedback for sometime, but it supports a teacher-centred mode of learning. I would suggest that students being able to critique each other and themselves is far more important.

  2. Some good points raised here Greg. We do need to be careful of balance here, as Kelvin comes from the end of the spectrum. The following is a link to a very well written critique out of Massey Uni,

  3. greg.carroll says:

    yup Ben agreed. But if you can wade your way through Hatties book then you can read a counter-argument too?
    I guess I am agitating for listening to those counter arguments and reflecting on them as well as the party line and mainstream.
    Avoiding the ‘echo chamber’ I have blogged about before where all we hear is our own beliefs coming back at us because of who we choose to listen too for our professional learning.
    Finding the middle ground between the technocratic and where Kelvin sits is probably the answer ….

  4. Steve Ladbrook says:
  5. Robin Sutton says:


    I am a keen follower of Hattie’s work (basic professional courtesy there, so you know which angle I come at this from..LOL)… but I don’t believe that Hattie himself would support some of the interpretations that he is being accused of by some of his critics.. close reading of his work indicates that he is VERY well aware of the limitations of his own research, he is a skilled academic. There is far too much of the Sunday Star Times type interpretation of his work, I’d class it as LAZY…

    This is a complex issue, probably the one statement we would all agree on.

    It reminds me of the learning styles issue.. i was once a keen LS advocate but, as John Church at Canterbury once said to me, there is NO replicable evidence to support the interpretations that some have peddled about that issue. This shattered my world, but pushed me to base my own practice on best evidence, rather than on myth and superstition.

    I wonder if those who have been most vocal in the criticism of Hattie’s work are those with the most closed minds?

    Just some thoughts… hope they don’t cause offence (because that’s not the intent)…


  6. Robin

    I agree with you re Hattie’s work. I have much time and respect for his efforts. I do own his latest book, visible learning. In the pages I have waded through to date he does discuss the limitations of ‘meta’ research.

    I agree mainstream media of the likes of Sunday Star Times, dumb down what can very complex issues of learning. However Hattie does not help himself at times as he is the master of what can be arguably quotable quotes. Which of course are what the likes of Sunday Star Times really like.


  7. Robin Sutton says:


    so true… on all counts


  8. greg.carroll says:

    Robin -- certainly no offense! Harder to get me upset than to disagree with my ideas. If we all agree then nothing comes of it all for any of us …. and certainly no fun 🙂
    I believe I was very privileged in my studies at Canterbury in the late 80’s to have John Church, Adrienne Alton-Lee, Graham Nuthall etc as lecturers. Their classroom based research and methodology have certainly had profound influences on my thinking.

    Hattie is the master of the sound bite for the media, and is way too smart to not know that what he is saying will be sensational enough to be picked up on and quoted. He is an accomplished researcher and a formidable intellect, agreed, but he also has agendas I believe, and we need to be judicious in what we follow and buy into along with the data.
    There is an argument for Hattie being as extreme in his assumptions and beliefs as Smythe and Hammonds? Vince Ham from CORE has a quote above his desk “if you torture data sufficiently it will confess to anything” -- I always remember that when looking at statistics, and particularly the conclusions others draw from them

    We all know from our experience in our own schools how aggregation of achievement data loses something. It is the individual children who are important not necessarily what statistics say they should be like. Or are like collectively rather than individually. Like all data it is the outliers who are challenged the most by generalisations we make whether they are based on research, statistics or our own prejudices.

  9. Robin Sutton says:

    I haven’t seen Vince’s quote.. LOVE it.. thanks He! He!


  10. Hi Greg

    Pleased you caught up with my blog re John Hattie. You may remember you once saying to me I was a little niave about Hattie’s ‘meta’ findings ( I was more generous towards him then ). Time will tell about Hattie’s influence ,for better or worse, but I have placed my bets! Nice to know Kelvin and I are at one end of the debate and Hattie at the other -- all new ideas come from the edge! And did someone say I needed to be more open-minded! If my mind was any more open I would catch a cold! I am more worried about the gullible out there! Did you read my recent blog about Herbert Kohl’s letter to Obama about Standards?

    Ka kite ano

  11. greg.carroll says:

    Hi Bruce …. it is the 15% of bees who go forth and find the new food for the hive. Without them the hive would starve. I think the same logic applies to intellectual endeavours and to education -- without the ‘radicals’ we don’t change and new ideas just die.
    Wasn’t it Twain who said something along the lines of disagreements making horse races?
    Haties smugness makes me suspicious of motives etc …. certainty and zealots always make me ask questions, particularly in this game.

  12. Hi Greg,
    I am interested in how you can find assessment used to create League Tables for nations stuff reliable/valid/acceptable -- when you find asssessment used to create League tables for schools unreliable and invalid --

    Is it the nature of the questions in the assessment?

    For example --

    What is your analysis of the nature of the test questions in PISA that means you find them acceptable as a measure of educational outcomes when you reject the proposed assessment questions as a measure of educational outcome in the national standards.

    Do you prefer the PISA questions to the assessment approach used in TIMSS and PIRLS? Refer And if so why?

    I suspect that instead of asking “What is wrong with being 4th overall?” … we should probably be asking What is right with being 4th overall? What does the PISA result tell us that is worth knowing?

    Two places to start looking for answers
    (1) Doug’s post on The Global Talent pool --
    (2) Tom Nicholson on the probability that PISA results in New Zealand overestimate outcomes because they are based on measuring educational outcomes for a cohort of students tested at 15yo
    A possible answer is the effect of exclusions in that many of the problem children in our schools have left school or been stood down or suspended by the time they are 15 years old. School Leaver statistics show that 14 percent of the lowest achievers have left school by the time they are 15. That’s a lot of children. The number could be higher than this because the statistics do not include those 15-year-olds who make up a sizeable proportion of the 20,000 or so pupils who are regularly stood down or suspended each year. So probably the PISA survey over-estimated the reading abilities of the 15-year-old children we have at school and made things look better than they are. The discrepancy is really puzzling. I have more faith in the PIRLS results because they are focused on 10-year-olds who can’t leave school and because PIRLS studies have been taking place for over 37 years whereas PISA is only the last 7 years. Lagging in the reading stakes in the Dominion Post, 8 February, 2008

  13. greg.carroll says:

    Hi Artie,
    My main point with respect to PISA is that international comparisons are often used as a justification for pushing changes in our education system. But if we have actually done quite well this is a bit mischievous and dishonest.
    Any sort of meta-analysis or aggregation of large numbers of slightly different data sets like PISA, Hatties work, school wide assessments, etc have inherent difficulties of not really comparing apples with apples. I think it was Greg Whitby who recently wrote a good post about the introduction of League Tables in Australia and how they show little we don’t already know. Does PISA show the same sort of ‘insights’? …. I’m not sure. Does it matter? …. not sure either.

  14. a meta data survey may prove some points but is it possible the effect sizes in your school or my school vary wildly in reality, therefore we are taking things to the letter of the law and should use research as a guide.

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>