Greg Carroll |

The iceberg of outcomes

Posted on November 22, 2016 by Greg Carroll

This is the time of the year when both good practice and legislative requirements have schools focusing on their achievement and other student outcomes data. We are looking to see where we have made the biggest impacts so we can celebrate this. We are also looking to see if there are groups of students, or areas of our programmes, or parts of our school that are not making the progress we had hoped for. School leadership teams often spend a lot of time crunching numbers and making the huge mass of collated information meaningful and enable colleagues, staff, teachers, Boards, and sometimes even students to make sense of it. The question is: Are we focussing on the right things?

When we are looking at data we need to consider some important factors

Student Management Systems (SMSs) are making the analysis process a bit simpler if they are used effectively, and if the information being stored and collated is numerical. But, often SMSs are simply repositories for demographic information and a place to keep test scores, NCEA data, and Overall Teacher Judgments (OTJs). A basic understanding of spreadsheets certainly helps make sense of data on scale, and means different theories can be explored and views of data used to try and understand what it really means.

Being “data literate” also means being able to choose appropriate presentation formats for the kinds of data being shared. A table may show everything, but can be confusing and significant things can be lost. A graph is good for showing differences, but attention should be paid to using an appropriate scale, for instance.

Lack of experience with data analysis can lead to making incorrect assumptions. For example: the assumption that a numerical difference is a significant one. By this, I mean, ensuring that the differences in the numbers could not simply be attributed to chance. If numbers are small then the so-called margin of error can be quite large. Think about political polls, for example, that often quote a margin of error of plus or minus something like 3.5%. This means that the actual results could be expected to vary by up to 3.5% bigger or smaller. This is with surveys of over 1000 people. In the school context, we may well be talking about samples of less than 10% of this size. With a sample of 50 or less, the variance of 20-30% in scores could, in fact, be expected simply by chance. This is particularly true if the confidence we have in the accuracy of the measure being used is not that high.

In the school context, we also know that comparing one year group with another is not comparing like with like. Different year groups can have completely different compositions and the students can vary wildly in their engagement, confidence, and ability in different components of the curriculum we may be assessing and tracking.

Positioning assessment data in the decision-making process

‘Data-driven practice’ and ‘data-informed decision-making’ have become real buzzwords in recent times. Both these things require consideration of the factors outlined above. They also require that we position assessment data in a way where it is not the sole determining factor in what we do.

In the same way that good assessment practice means a single-test score is not the only indicator of an OTJ, analysis of OTJ data is not the only indicator of schools achieving successful outcomes for their students. Or, indeed, of teachers being successful in their settings either. Any assessment should be a point-in-time litmus test of the outcomes being aimed for, not the only criteria. Effective schools and individual educators know a lot more about their students collectively and individually than can ever be captured in a single number, or set of numbers. Student outcomes over time are not always well represented on a graph.

I like to think of the things we can put the number on and, therefore, ultimately turn into some sort of graph or table as the proverbial tip of the iceberg. There are so many other things that make up student achievement, outcomes, and success that are ‘below the surface’, but nonetheless hugely significant:

These factors below the waterline are things that the ‘tip of the iceberg’ factors can point to, but often the link may not be a very strong one. They may also be things that the whānau or culture that your students (or a group of students) come from are valued more highly than those above.

As a parent, I am way more proud of my own kids being good people than I am of any of their academic outcomes. I would think many families take a similar perspective.

So, I guess my challenge in this blog post is to consider several different things. As we begin bringing our focused attention onto the year’s data and information to begin making decisions about where we need to focus for next year in our programmes and improvement efforts:

Are we examining data in an appropriate way?
Are we reading too much into the numbers?
Do the numbers show what we are claiming they do?
Are the important things captured in the numbers, or, are there other key things that cannot be shown by numbers alone?
Are we using the best data and information that you can in your decision-making processes?
Are the conclusions we are drawing true for all students and groups of students?

If you would like support thinking about these things more deeply, and/or planning your PLD response to what you have found, do contact us at CORE Education.

Image sources

Iceberg photo (top and featured on home page): Image: By AWeith (Own work), CC BY-SA 4.0 via Wikimedia Commons
Iceberg of outcomes graphic: by author under CC-BY-ND-SA

Posted in

Memes for change. Getting out of the CAVE.

Posted on August 19, 2016 by Greg Carroll

The Internet is a wonderful thing. It has given us everything from dancing babies to cat videos. It is also an astounding place for the creation of communities of practice, the sharing of professional knowledge, and for people who are interested in quite narrow topics to find communities of affinity with others around the world.

One of the quite interesting things to come from the rise of social media in particular is the meme:

‘A meme (/ˈmiːm/ meem) is “an idea, behavior, or style that spreads from person to person within a culture”. A meme acts as a unit for carrying cultural ideas, symbols, or practices that can be transmitted from one mind to another through writing, speech, gestures, rituals, or other imitable phenomena with a mimicked theme. Supporters of the concept regard memes as cultural analogues to genes in that they self-replicate, mutate, and respond to selective pressures.’
Source: https://en.wikipedia.org/wiki/Meme

Memes are exemplified by the fast spreading of things, like the dancing baby above, through social networks and social media. The wondering for me is:

“How do we make effective leadership and school change practices into memes?”

How can we make effectiveness and high performance contagious and for them to spread like some sort of disease that has no cure? How can we develop the super-bug of effective school change?

Some of the key influencers of the effectiveness of change that I see in my work with schools and in the literature are:

Relationships — people need to develop trust and agreed ways of working before the ‘hard stuff’ of change can be a focus. Only surface and ‘safe’ things can be explored. There is a saying that equally applies to change:

This challenges us to ensure we have agreed ways of working and high levels of professional and personal trust before we can get to the really meaningful conversations and change efforts.

Understanding — I have written before about getting ideas outside the echo chamber and avoiding the Medici Effect. Are you really getting to the underlying cause, or simply looking at and dealing with the symptoms? I think we often do this with assessment data – perhaps our literacy results are lower than we would like, so we do literacy Professional Learning and Development. But if we dug a little deeper and explored the information available to us a little more, we may find that engagement or home-school partnerships are larger causal factors than teacher pedagogical content knowledge specifically in an area of literacy. Sometimes, we don’t really understand the underlying causes of the data we are seeing, and, therefore, put our intentional change efforts in the areas that will not give us the best ‘bang-for-buck.’
A focus on comfort and happiness — change is often difficult and uncomfortable. If the index of success is the degree to which everyone is comfortable and happy, then change becomes impossible. Comfort and change are, at least to some degree, mutually exclusive. Understanding this (perhaps through models like Fullans’ Implementation Dip, or Nottingham’s Learning Pit) is a key element of beginning change processes and ensuring change and improvement are continuous and ongoing processes in a school.
Planning and Checking — we often make changes without a clear set of measures for what ‘good’ or ‘effective’ looks like, or what we are expecting to see as outcomes of the initiative. We know we want things to be better than they are now, but without agreed measures how will we know if we are on track or have achieved them?

So we do know what some of the key influencers of change are. We know what works! We also do know what some of the highly effective processes are. We do know what highly effective practices and outcomes look like. So this begs the question — ‘Why are we not able to pull things together to achieve the package that makes the gains for students which we are all hoping for’?

Most of the things I see answering this provocation come down to people. How we work together; our mindframes around achievement; and more particularly our mindframes around change.

To simplify things a lot, people tend to sit somewhere on a continuum with respect to change: From the CAVE mentality to those with a high BIO quotient.

So my pondering becomes how to make the BIO mentality sticky and contagious. How to get people onboard with change. And conversely, how to get those who live in the CAVE out into the daylight and revelling in the excitement and challenge that highly effective school-change brings.

I would love to hear what people have found works well in your context! A big part of spreading a meme is enabling the contact and having the conversations. Let’s share our effectiveness ideas in the comments below.

Image sources:
Maslow quote: Future Classrooms (slide) Dean Shareski CC
Diorama of cavemen — Wikimedia – Public Domain

Posted in

Avoiding the snake oil

Posted on June 8, 2016 by Greg Carroll

I have written previous posts about how we collaborate, prioritise and ‘make and break’ decisions about school and organisational change. The decision-making process and how we seek different influences to inform it continue to fascinate me. There is no one answer or simple prescription that can be followed in these areas, and, consequently, one thing many schools struggle with is how to allocate the very scarce resources that they have. Each school has only a limited amount of ‘discretionary spend’ in financial terms or in the focus, time and attention of the staff. There are a lot of competing things we could focus on for our professional learning or spend our curriculum, property or staffing budgets on. The question is, how do we make sure we are getting the most positive outcomes possible?

When I was a principal, for example, much of the mail that came across my desk was offering deals on this product or that programme, and contained sometimes quite extravagant claims about the results that could be expected for our staff and students. You have to have a well-attuned filter at times. There are plenty of snake oil salesmen out there who will push hard for you to spend sometimes significant amounts of that discretionary spend on their products or services, when they may not align well with your articulated beliefs. So, how do we make the decision about which things are, in fact, the most important and influential levers for positive change in the outcomes for our students?

Posted in

Red Teaming school change

Posted on March 1, 2016 by Greg Carroll

I wrote a while back about getting out of the ‘education echo chamber’ and challenging ourselves with people who may think differently or come from a different perspective than us. One of the links in this post was to RedTeams.net, who are counter-insurgency and security specialists. I find the whole concept of ‘red teaming’ fascinating, and those I work with will often hear me talk about trying to ‘break ideas’ and thrash plans around while they are still in their formative or concept stage. Red teaming has evolved from the historical Vatican concept of the so-called devil's advocate – someone whose job it was to try and break ideas or plans, or to argue from the opposite perspective to the status-quo or accepted doctrine. The counter perspective was seen as essential to coming to good decisions and decision-making, just as actively seeking multiple perspectives is central to effective change in education contexts today.

One of my current favourite reads is Micah Zenko’s ‘Red Team — how to succeed by thinking like the enemy’. It outlines the whole concept of Red Teaming, and gives examples from a number of different fields across military and business arenas. He begins by observing:

Institutions — whether they are military units, government agencies , or small businesses — operate according to some combination of long-range strategies, near-term plans, day-to-day operations and to-do lists. Decision-makers and other employees do not simply show up to their jobs each morning anew and then decide then and there how to work, and what to work on. The existing guidance, practices, and culture of an institution are essential to it's functioning effectively. Yet, the dilemma for any institution operating in a competitive environment characterized by incomplete information and rapid change is how to determine when it’s standard processes and strategies are resulting in a suboptimal outcome, or, more seriously, leading to a potential catastrophe. Even worse, if the methods an institution uses to process corrective information are themselves flawed they can become the ultimate cause of failure. (pg: xvi)

To me, that sounds very much like the dilemma schools and centres face each and every day, particularly at this time of the year as they are refining and confirming their strategic planning and day-to-day ways of working for the new year. It also reflects closely the understandings we have about the work we do in Learning with Digital Technologies to support schools and clusters to implement their plans and goals, but with a specific e-learning lens. Planning, making strategic choices, change management and ensuring the smooth implementation of actions promoting change towards agreed outcomes are all crucial elements of what we support schools and clusters to do.

Zenco (in Chapter 1) also describes six critical factors to the effectiveness of any Red Team programme. Once again these things will sound very familiar to anybody who is involved in school leadership or change management. With a specific school or centre context, these factors could look like:

The boss must buy in: The support and engagement of the leadership in the entire programme and its outcomes is the most critical single factor in schools, centres and for Red Teaming.
Outside and objective, while inside and aware: Those leading or supporting any programs or changes must be aware of the culture of the organisation and effective ways of engendering change within it. They must also understand who the official and unofficial leaders are and who it is most effective to work with and through to get the desired outcomes.
Fearless sceptics with finesse: Don’t make assumptions — check them, break them, challenge them, and change them. Dance carefully around and between the things and people that may be blockers or impediments to change. Work carefully and skillfully with those people who may not be as on-board as others.
Have a big bag of tricks: If one strategy doesn't work effectively, good change leaders always have other ways of getting things to happen. They will know who the effective people are to collaborate with, and what strategies are most useful to work with them to get the change they desire.
Be willing to hear bad news and act on it: Once again, effective leaders will be constantly reviewing and checking that they are on track for the outcomes they are seeking. They will be prepared to make changes along the way, and, if necessary, refocus their efforts in ways that will promote the long-term outcomes and gains they are seeking.
Red team just enough but no more: You can over plan! At some point you need to get on with it and implement change, not just plan it and think about it. Fullan often quotes an inverse relationship between the overt ‘quality’ of strategic planning and the ‘quality of the outcomes’.

Redteams.net has a moto of “Plan, execute, vanish”. Again this has a strong education parallel:

Plan well and for all contingencies.
Do what you planned, and said you were going to do.
Re-focus your change management attention and get on with the next thing when you have achieved your goal/s.

So, as you reflect on 2015, and really begin to ramp-up your school or centre development in 2016, what elements of Red Teaming can you include?

Do you actively try to break goals and plans while they are at the formative and intellectual stage so you are less likely to be surprised by something you never thought of?
Do you actively seek out the wacky and weird ways things that may go wrong – because they often do?
Do you seek the perspectives of those you disagree with or ask the dissenters for their ideas?
Do you over or under plan?
Are you even planning and actioning the right things, the ones that will have the greatest impact?
Do you over-labour things and not move on to the next thing you need to do?
What will you do if your plan does begin to falter? Can you bring it back on track because the challenge you are facing is something you have already considered?

What other questions do you need to ask yourself to ensure that you achieve the things you aspire to for yourself, you students and your school/centre this year?

Posted in

Doing what makes the biggest difference

Posted on November 4, 2015 by Greg Carroll

applying the 80-20 rule

Source: Adapted from photo by John Nakamura Remy under CC

I have been doing quite a bit of work in schools, recently, who are right at the very beginning of their e-learning journey. For these Schools, things like GAFE (Google Apps for Education), Flipped Learning, and lots of the other things that many people reading this may now take for granted are, in fact, really new. One of the first things people in this situation tend to ask me is, “What should I do?”

Let's reframe the question

What I think we should really do is reframe that question around impact. I, therefore, often answer the question with a question along the lines of, “What do you think the one change you can make is that will have the biggest positive difference to the learning for the students you work with?” Another addition to this could be, “ … and for the least amount of change and/or work for you”. I often quote the 80-20 rule, sometimes known as the Pareto Principle, when having these discussions. The premise of this is that 80% of the results or impact come from 20% of the energy or change. In the pedagogy context this could be thought of as identifying what the initial change/s are that will make, or have, the biggest impact on the students.

Start small and get it right

It is essential that any changes made rapidly become embedded in practice and part of the ‘normal’ in the classroom or school. Starting in a small and focussed way and using an Inquiry mindset, is a useful way of doing this. You do one thing and get it right, then quickly move on to the next. Don’t revamp everything in your programme at once. Start small and learn what you need to know to make that thing successful, then scale this learning across more of your programme and pedagogy.

For example: change something with one reading group, get it right, and then roll this out to everyone. Experiment, do it well, and learn from the implementation of this prototype about what needs to happen when you scale things up. Once you have sufficiently mastered things, move onwards and upwards to include more and more people, and more and more of your programme.

Bells and whistles don't always make things better

One of the real challenges of technology and the digital space is the highly seductive nature of it all. I have seen many schools and teachers fall into the trap of trying to ‘do it all at once’ and turning their schools or classrooms into a digital christmas tree with all the whistles and bells at a meteoric pace. The sad bit is though that the whistles and bells may not in fact be making things any better for students, or the learning any deeper. Sure the teacher is working harder and things are taking longer. Kids are having to learn new routines and ways of doing things. But a lack of focus may actually be detracting from the learning. The whole thing quickly becomes unsustainable for both the teacher and the students who were supposed to be benefitting from this in the beginning.

Finding the sweet spot using the 80:20 rule

Now, don’t get me wrong, I am definitely not advocating for inappropriately slow implementation of change any more than I am for over-paced development. The ‘sweet spot’ is somewhere in between.

I was reading about this idea recently, which takes the 80:20 rule above and extrapolates it out even further:
“What if you took your 80-percent results and applied the 80/20 rule to them? And then one more time?

80-20 rule graph

Source: ShawnBlanc.Net

What you end up with is the idea that your initial 1-percent of energy spent brings about the first 50-percent of results.

It's about finding the key things that make the difference

For me, this hits home the point about key things making the difference, and choosing that sweet-spot where we get the best bang-for-buck in terms of both simplicity and impact. Small changes often have disproportionately and deceptively big impacts.

I wonder then, if we can come up with a set of questions we can ask ourselves in order to force a focus on some fundamentals and the things with the biggest influence?

Here is my attempt at this:

Question	Rationale
What (potentially) small changes COULD I make that will have the most significance for the learning of the students I work with?	Trying to focus in on the key small actions or changes rather than the big ones — eg: introduce a choice of digital activities for the non-instructional time with the XXX reading group NOT digitise my entire reading programme
How much personal learning and change is required for me, for my programme, and for students?	Is the learning curve involved for the teacher and the students worth the time and effort? Remembering that learning software is not the focus of our programmes and the purpose of our time in the classroom is not to be trained in apps, tools or technology.
Is the proposal doing things better or just differently?	Are we exploiting the potential of technology to make things possible that are not in a ‘pen and paper’ or analog world? If not then is having the tech the best option?
Does the ‘better’ justify and outweigh the ‘different’?	If the answer is yes then this proposal may have real merit
Of the possible changes I COULD make, which SHOULD I make to have the biggest initial impact?	Choosing the most ‘impact-full’ option among the possible choices as the place/s to start.

What other questions or considerations would you include?

Posted in

CORE Blog

CORE Blog

The iceberg of outcomes

When we are looking at data we need to consider some important factors