The OECD report is a bit of a tome and pretty heavy reading but this quote just struck me:

“Decisions regarding the subject areas and grades or year levels in which student assessments will be used … are particularly important as they delineate the aspects of a school upon which performance is measured. These decisions therefore define what is meant by a school when estimating schools value-added scores to promote school accountability, school choice, or school improvement. If students are only assessed in mathematics and the language of instruction then the definition of a school is those aspects of a school that contribute performance in those measures in the grade or year level in which the assessments take place and, depending on the structure of the school system, the grades and year levels leading up to the assessment.”

OECD pg 177
Do we want our choice of testing to define for us what a school actually IS? This goes further than I would have.  It confirms that what we are nervous about may very well come to pass with the aggregation of the National Standards data with the MoE.  Do we really want our the definition of the concept of ‘school’ to be the narrow range of skills covered in the Standards.  Ken Robinson et al would have a FIT!

The report is also expressing a lot of caution about using RAW achievement data to target school development initiatives.  The say not correcting for those factors outside the influence of the school makes drawing any valid conclusions extremely difficult (p14, 28).  This was one of the prime motivators we were told for the implementation of all this in our ‘consultation’ meeting, to enable the MoE to identify those schools requiring support and target resources to them.

Clarity of purpose is key. (p178)  Purpose dictates the measures, how they are evaluated and the kind of data that is gathered and how it is analised.  Are we clear on the purpose of the National Standards? The report states that if the purpose is accountability then a single measure is perfectly sufficient.

I am absolutely certain there is a will with the MoE to make the National Standards the best they can be and for all the thinking around them to be worked through.  Trouble is the haste with which all this is happening means this can not be done well.  We will be implementing a system with holes in it that we will only find along the way.

In our change management processes at school we work hard to brake any model at the conceptual and theoretical level before we put things into practice in classrooms and with the kids.  I am not sure we are enabling this same process here ….

Cohort size also makes a difference and it is noted that small schools in particular produce unstable and more unreliable data:

“participating countries generally considered cohorts of less than 20-30 students produced …. estimates that led to problematic interpretation of results” (p181)

This is simply a function of having a smaller group to moderate etc rather than the abilities of the teachers.  So we are saying we need a school of approximately 150 to produce valid data …. mmmm.  Bit of a problem for a country where this is about a third of schools.

Also an issue with correcting for the “outside of school influence” factors when we have been assured individual children will not be identified. (p183)  All we end up testing without this is home background (p ??)

I guess the key message is that any system is only as good as the planning at the outset and the thinking that is put into it.  Also the data quality and integrity is crucial and needs to match the PURPOSE.  All things to be worked through for the National Standards.  Interestingly too there is an absolute assumption that any programme will involve a pilot programme (p193-5), not something I have heard mentioned at all.

Page 198 recommends no results be published until trends over three years can be established to ensure that schools are gathering and reporting valid and stable data.  Interesting!!  “Extensive” consultation to engage stakeholders and training “is required” (p199)  So to is training.

2 Responses to “testing defines the concept of the school”
  1. Just wanted to say how much I appreciate you sharing your reading and research, thoughts and opinions around the National Standards issue.

    My contribution to the debate lies purely in my experience as one who taught for 5 years in the UK before emmigrating.
    Having schools driven by national testing, with league tables of said schools defining schools, for better or worse creates a terrible culture of staff and students alike.

  2. greg.carroll says:

    Thanks Simon,
    I believe that behind the scenes a lot of thinking and debating is going on with the minister and MoE. The public face has to be politically stoic but this is our one and only opportunity to have any influence at all on the outcome of the policy.
    Frustrating but we always need to be well informed and aware of the wider educational landscape and not simply the one that ends at the school gate.
    cheers
    Greg

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>