Category Archives: Assessment & Evaluation

A renaissance in assessment?

The end of year is a time where the focus of most teachers and students is on the summative assessments we incude in our education process. These forms of assessment have been with us for decades, and changed little despite significant change being experienced in other areas of our system, including pedagogy, technology etc. The video above is one of this year's Ten Trends from CORE, which makes the case for new apporaches to assessment

Assessment14The argument for re-thinking assessment is not new, but while it has been debated for some time, there has been little done to act on this thinking. "The existing exam system is unable to distinguish between test performance and knowledge and does not lead to genuine progress", according to a leading educationalist reported in the Times Education Supplement (TES)

What we need is a massive re-think, a revolution in our approach – and a new report titled "A Renassance in Assessment" authored by Sir Michael Barber and  Dr Peter Hill provides a thought provoking response. The report is critical of current testing regimes for focusing on a narrow set of low level skills. The authors criticize an over-reliance on grades that reveal little about what the student can do and call for “validated learning progressions” with efficient processes for collecting and analyzing data and easy-to-use assessment tools.

In contrast to current state testing systems, Barber and Hill suggest assessment should:

  • Accommodate the full range of student abilities
  • Provide meaningful information on learning outcomes
  • Accommodate the full range of valued outcomes
  • Support students and teachers in making use of ongoing feedback to personalize instruction and improve learning and teaching
  • Have integrity and that are used in ways that motivate improvement efforts and minimize opportunities for cheating and ‘gaming’ the system.

There's a lot of excellent reading in this report – much of it supporting what I've noted in the Ten Trends video, and also reflecting the thoughts of other educators such as Fullan and Hammond, Wilhoit, and Pittenger, and echoing the content of a report on the Opportunity to Lead released in March this year. 

An important thing to mention here is the role that technology will play in our future thinking about assessment.  It's great to see the Qualifications Authority in New Zealand making moves towards online assessments, something we need to support and encourage – but there's a danger here that it could simply be seen as a substitution approach, with online tools replacing the traditional pen and paper for exams. 

Barber and Hill recommend the use of adaptive testing to generate more accurate estimates of student abilities across the full range of achievement while reducing testing time. They note that online environments facilitate, “The collection and analysis in real time of a wide range of information on multiple aspects of behavior and proficiency,” as well as, “More immediate, detailed and meaningful reporting to specific stakeholder groups, such as via smartphone/tablet devices and through the creation of e-portfolios.” 

I believe that 2015 may well be a year in which we see some bigger steps taken towards the renaissance in assessment that this report describes. I'll leave you with some of the quotes from the report to whet your appetite…

assessment quotes

Introducing PISA

In New Zealand it seems we are constantly being reminded of our rankings in the international PISA results, particularly where disadvantaged groups of students are concerned. While the figures and comparisons get bandied around a lot, what's interesting is to understand how they are arrived at, what goes into collecting and representing the data that is shared.

The OECD have just shared the video above in an effort to help us understand why the OECD’s number-crunchers trigger such intense debate about the state of education around the world every time they release the results of the latest PISA survey. Worth a view if you've got a spare 12 minutes and a cup of coffee.

Pedagogically driven…?

Over the past few  years I’ve frequently heard the comments; “It’s not about the technology, it’s about the pedagogy”, or in relation to the advent of ultrafast broadband; “we’ve got to drive it from the teaching and learning.” These are well intended sentiments, but why is it that the technology still dominates much of the discussion, and so often becomes the starting point by default? And just what do we mean by letting the teaching and learning lead? How would you explain that to someone outside of education – or, for that matter, someone inside?

I’ve been thinking about this for some time now and this week I had a chance to share some of my thinking with Russell Burt, principal at Point England School. I’ve captured our jottings at the whiteboard in the diagram above (click on it for a larger version). Key points are:

The fundamental activity of students as learners is shown in the ‘process’ line, consisting of:

  • Learn – This is the part of the learning process where learners are exposed to intentional teaching, access to content, the development of pre-requsite knowledge and skills etc.
  • Create – this is where learners work with, manipulate and re-present the information they have gathered in the ‘learn’ scenario. This may be done independently or collaboratively, and may involve a range of tools and environments.
  • Share – where learners communicate with others what they have learned, being especially aware of the audience. May involve a celebration of some sort, and feedback where appropriate.

These three things are underpinned/supported by a comprehensive assessment process – both for and of learning. The assessment band is deliberately shown this way to emphasise the fact that it isn’t driving the learning, but is an integral part of the learning process.

Below that are the three principles that we think are key to developing an effective, future-focused approach to teaching and learning:

  • Ubiquity – focusing on learning that happens anywhere, any time, any pace and through/with any device. Learning is no longer confined to the ‘box’ of the classroom, the ‘fences’ of the school yard, or the period between 9-3 in a school day. It also reflects the underpinnings of life-long learning.
  • Personalisation – learners in charge of their learning, with the ability to make meaningful choices about all aspects of what and how they learn. Also recognises the imperative on teachers, as learning designers, to recognise that all learners learn differently and in ways personal to them. 
  • Collaboration – recognising that the ability to work with others is an essential skill in a future-focused world. Acknowledges the theoretical underpinnings of social constructivism and connectivism, and the need to recognised and work with each other’s strengths (and weaknesses).

The layers below this illustrate the way the services and infrastructure supports our intended learning outcomes/teaching approaches. Working from the top, it becomes relatively straight forward to see how the teaching and learning may drive the technology decisions. Take this example for instance:

Beginning in the ‘learn’ area, a school decides it is important that students have unfettered access to a broad range of online resources and materials to support learning in the classroom as it is required (just in time).

Acknowledging the ‘ubiquity principle, the decision is that access should be available anywhere on the school premises, but shouldn’t be confined to the school day and school premises.

This leads then to decisions about mobile, internet capable devices, school-wide wireless, cloud-based content servers and filtering solutions that aren’t unnecessarily restrictive.

Although this isn’t a particularly detailed explanation, and there are a range of other factors and possible scenarios that could be developed to meet the original ‘learning’ need, you’ll get the point. The important thing is that the model is an attempt to initiate some dialogue about how we articulate our teaching and learning needs, and how they can then be appropriately supported by the technologies available to us.

Ken Robinson on standardised testing

Sir Ken Robinson on standardised testing – he doesn’t appear to be too great a fan.

(sorry, can’t work the embed function from this site)

Assessment using technology

A brief but informative booklet has just been released by BECTA titled Messages from Evidence: Assessment using technology. A quick read of only 7 pages, the booklet works from the premise that technology has an important role to play in facilitating assessment for learning. For instance, technology can be used to administer a test and then link each learner to a suitable learning task, based on test results. (This is a key part of the design of e-asTTle, and other programmes such as Mathletics.) Also, using technology can help learners assess
each other’s work (blogs are commonly being used in this way at present, along with some custom developed software), and it can provide teachers with regular reports on the progress
of all learners (feedback within some LMSs and some SMS packages do this).

Early investigations, however, revealed that few teachers are making use of technology for assessment. This is, in part, because some teachers aren’t aware of the role technology can play in increasing the impact of their work with learners. Some teachers are not sure about what they should do with technology-enabled assessment information once they have it. (click the graphic to the right for a larger image)

The booklet identifies four key challenges for schools:

  1. using technology to integrate assessment into everyday learning
  2. getting hold of high quality assessmetn tools
  3. making assessment information available online, for example, for parents
  4. passing assessment information between institutions, for example at transition to secondary school.

The booklet then provides a number of succinct case studies as examples of what some schools are doing to address these challenges. If you’re looking for specific examples of assessment tools then you’ll need to look elsewhere – this leaflet is more of a high level monograph, painting the broader context for considering using technologies for assessment. The table on page 4 is particularly useful, and could be used at a staff meeting or at department level to begin thinking strategically about the ways assessment is currently being carried out and recorded, and the ways technologies (those that exist in the school already, and others that may be accessed online etc) can enable this to happen.

A handy weekend read, and sufficient stimulus in it to provoke thought and further action.

CORE’s ten trends for 2010

Today I presented CORE’s ten trends for 2010 to an audience of around 400 delegates at the Learning@School conference in Rotorua. The ten trends are a collection of themes and issues that have been identified by CORE staff as trends in education that we imagine will impact on the work of teachers and leaders in early childhood centres, schools, and tertiary institutions in NZ in the coming year. While our focus is on the bigger picture of education, there is a focus on trends associated with the use of ICTs in education, reflecting the fact that we are living in a world where nearly everything we do has a digital dimension.

Measuring the right things

I’ve just read a fascinating publication from Microsoft titled “Interoperability: Improving Education” which came about as a result of 10 or so educators and ICT practitioners who were brought together by Microsoft for a meeting running alongside the annual NAACE conference held in Blackpool, England, earlier this year. The brief was to talk about the way that schools use pupil data. And the wisdoms that ensued are contained in an new Microsoft discussion document for school leaders and local authorities, “Interoperability: Improving schools” (download the PDF here).

The contents of this paper provide timely insights for NZ educators because it nicely ties together two pieces of work that are currently the focus of the Ministry of Education. First are the discussions about standardised testing, and the measurement of student progress and achievement, and second are the issues of interoperability as they relate to work being done around data interoperbility between LMSs and SMSs in schools, and the whole area of e-portfolios.

From its title, and the fact that it’s published by a technology company, you could be forgiven for thinking that the document is about a technology solution that will end all our woes. This is not the case. Instead, the document contains a summary of thoughts in response to questions such as:

  • are we collecting the right data?
  • what data should we be collecting?
  • who needs, or wishes to see the data?
  • can we easily move data to where it’s wanted or needed?

The context for the discussion is identified in this excerpt from the introduction:

The last five years has been dominated by discussions about common file formats, and competing systems which support data interchange standards. The International Standards community, through its work on file standards, has helped us reach a situation where students and teachers can easily share assignments, examination submissions and documents.

The same cannot be said for simple data interchange – for example, the simple requirement to automate the process of keeping a list of users up to date within a learning platform without a manual intervention. And those solutions which do exist for this appear to need customisation for each data relationship – between different learning platforms for example.

I believe that we are collecting the data within our educational systems that we need to deliver
improvement, but only those data items that are being seen as “part of the system”. We collect a core of formal learning data in our schools Management Information Systems, and through other systems we collect further datapoints – often disconnected from the core learning data – on health, achievement and engagement.

In the 21st Century, we are seeing a huge growth in learning and engagement outside of the formal education system. As we continue to build extensive connected learning communities, we need to find ways to see the holistic story of a truly connected learner, including their learning in school, in the community and individually. We need to move from a top-down data culture (ie we measure what the managers above us want measured) to an individually driven data culture, where the individual has more input to the data that tells their individual story, and where their past learning journey is used to support their future learning journey.

To achieve this we need to think outside of the strict confines of top-down, organisational data collection. We need to ask questions from a different perspective “How can students self-asses their skills and use that to improve their learning?” and “If we asked a student to tell us how they are doing at school, what data would they share with us?”

I am currently enjoying being a part of separate discussions (online and offline) around each of the issues identified above – but perhaps there’s good cause to reflect here and think about how timely it might be to work like this think tank, and engage in some robust discussions that actually  link both parts of the equation?

Perhaps it would allow us to reach similar conclusions as the English did as a basis for moving forward:

The answer, as it emerged in discussion, is that there‟s arguably too much emphasis on one kind of data. The current pattern of top-down accountability, it‟s suggested creates an emphasis on classroom attainment at the expense of skills and competencies. Or, as Sir Mark Grundy puts it,  “We’re measuring the wrong things.”

Technology and the future of student assessment

“Students today are growing up in a world overflowing with a variety of high-tech tools, from computers and video games to increasingly sophisticated mobile devices. And unlike adults, these students don’t have to adjust to the information age—it will be all they’ve ever known. Their schools are gradually following suit, integrating a range of technologies both in and outside of the classroom for instructional use. But there’s one day a year when laptops power down and students’ mobile computing devices fall silent, a day when most schools across the country revert to an era when whiteboards were blackboards, and iPhones were just a twinkle in some techie’s eye—testing day.”

This report out today from  Education Sector Research is a provocative read, and worth taking the time to digest. As highlighted in the intro above, the report points to the growing disconnect that exists between the increase in use of ICTs in classroom teaching and learning programmes, and its absence from the area of formal assessment.

The writer argues that technology has the potential to do more than just make our current approach to testing more efficient, pointing to new research projects that are demonstrating how ICTs can both deepen and broaden assessment practices in primary and secondary education, by assessing more comprehensively and by assessing new skills and concepts.

This is not a “once-over-lightly” analysis – it provides a useful overview of key issues and points to some promising models.  The writer provides some background to the use of ICTs in assessment over the past decade before illustrating the potential of ICTs in assessment with examples from the current research projects before exploring concepts of assessment linked to cognitive development etc. What particularly stood out to me is the conclusion that the most useful contribution of ICTs in assessment is in formative assessment, providing descriptive data that give teachers better information about how students are progressing and why they are performing at their current levels. They also provide insight into what changes to instruction might be the most effective (something I’m sure the users of e-asTTle will be pleased to have reinforced 🙂

Download “Beyond the Bubble: Technology and the Future of Student Assessment”(920Kb PDF)

Taking formal education beyond exams


Interesting article in the New Straits Times here in Malaysia over the weekend titled Taking Formal Education Beyond Exams. IT features and interview with Datuk Dr Adi Badiozaman Tuah, the Malaysian Examinations Syndicate director in which he outlines his views on the need to review the focus of assessment in Malaysian schools. In the interview he states

    “Current practices see us focusing only on the mental capabilities and cognitive domain of our children. We want to go beyond the paper and pencil tests and look at other domains to show the individual’s progress and development”

    The director wants to create a new assessment system that will shift the emphasis from public school examinations to more regular school-based evaluation of students.

    Having spent time here on a few visits now working in local schools I can understand why this reform is being considered. There is a high emphasis here on achievement in tests and exams, which creates a focus on ‘covering what is in the curriculum’, which itself becomes extremely prescriptive. As a result, there is little leeway for the development of creativity, innovation, critical thinking etc. – or for pursuing topics/themes or ideas that may emerge in the course of the day’s study and interactions.

    From the perspective of attempting to integrate ICTs into teaching and learning, this limits the opportunity to exploit the creative, expressive and communications potential of the technology and sees it become more of an electronic text book and marking device.

    Of course, this is not just a problem in Malaysia – it’s a perennial problem for educators the world over – wherever there is an empahsis on high stakes testing. Thus I am encouraged to see the Minister here announcing plans for a reform at a national level. Also encouraging to see a newspaper devote so much space to such an article – the full interview took up a two-page spread in the March 4 edition.

Ten Trends for 2007


We’ve added a new logo on the right hand side of the CORE website -titled CORE’s Ten Trends for 2007

Clicking on this logo will take you to a list of ten trends that we’ve identified as being particularly important in 2007. The aim is to create some dialogue around some of the things that are happening in the NZ context regarding the use of ICT in education. The emphasis is on looking at the bigger picture, rather than the things absorb our time every day at the “coal face”.

I will be using these ten trends as the focus of my Spotlight at the Learning@School conference this week, and would love to see plenty of contributions being made in the comments section at the end of each “Trend”.

Each month or so the CORE staff are going to expand one of the themes with further links and references to prompt a deeper level of participation and discussion. We’re sure to have missed some that people think are important, or included some that others think aren’t – all of which should make the discussion fuller and richer!