Making a start with student data analysis

Background

In my last post, I unpacked the Learning Management System (LMS) and the Learning Record Store (LRS). In it, I introduced the idea of capturing fine-grained student data using the Experience Application Programming Interface (xAPI) to inform programme, course, and module design. I felt it may have left many readers with a bit of a tall order, and in need of a simpler place to start.

That was the case for me back in 2010 at a regional polytechnic. With classes of 30 students, lectures to prepare, assignments to mark, at the same time developing an online community and resource, to also try to go deep into user data analysis would have been a bridge too far.

In this post, I plan to offer a starting place; some low-hanging fruit. It’s a complete starter pack for someone who has a course in Moodle and no idea where to start. I am sure it would easily transfer to any mainstream LMS.

Identifying students at risk

In your role as an online teacher (tutor, instructor, facilitator), the first thing you will want to do is identify those students who are at risk.

There is no greater risk for a student than not being present. In a physical classroom that is obvious; there are some empty seats. On the first day of the class, Sheryl and Tania are missing. The teacher wonders if they are aware they should be in this class. Perhaps they did not read the notice that said the room had changed. Perhaps they have discovered a clash with another class. After the class, the teacher has a duty to follow up and find out where these two girls are.

The same thing happens in the online world, but all too often the absence goes unnoticed. The teacher’s attention is focused on the people who are in, say, the introductory webinar — whose sound is not working, who have too many distracting questions. Especially if the enrolment is large, Sheryl and Tania may not be missed.

There are ways to resolve the problem, like recruiting an assistant facilitator, but that’s not our focus here; I was using it only by way of illustration. Let’s focus on Sheryl and Tania and use the tools available to us to determine if they are at risk.

By clicking on the link in the People block, Moodle displays not only a list of the people in the course but also when they last logged on. If it says “NEVER” beside Sheryl and Tania, then you know you have two students at risk.

If that was all an online teacher ever did — follow up on the NEVERs — that would be good in itself. But, if you have an ounce of geek in you, then there is a whole lot more that you can do.

Exploratory data analysis

The thing about exploratory data analysis is… it’s exploratory. Think of your data as being like a territory, and you set out on foot to get to know it.

Whether you are a beginner, a journeyman, or an expert in data analysis you will always start by just considering the data. Look at it, shuffle it about a bit, make a cup of tea, ponder. Don’t rush in, you may miss something that is staring you in the face.

For example, look at these five email addresses:

john.smith@stjosephs.school.nz
mary.roberts@stjosephs.school.nz
lbooth@middlebeach.school.nz
quackers@hotmail.com
garyb@chchpoly.ac.nz

I won’t say anything for now, but we’ll come back to this list later in the post.

John Tukey

John Tukey is the father of exploratory data analysis, and he wrote a textbook of that title. Despite all the advances since he wrote it in 1977, it is still a good place to start your journey. If you are the textbook type, that is. You may prefer to simply learn by doing.

Let’s do a little Tukey-type thing now. He talks about, ”scratching down numbers”, and this method is called stem-and-leaf. It gives you a quick visualisation and, therefore, the beginnings of an understanding of the numbers.

I’ll make that a little easier to see:

15	9
14
13
12	6
11	4
10
9	9
8	113
7	00953
6	3208473
5	501479001
4	622712541
3	3741011679
2	63224
1	3217
0	9800

Let’s say that you’d like to quantify a feeling you have that your cohort of 60 students is contributing quite well to the discussion forum. The stem to the left of the line is the tens column; to the right the ones. As you can see in the image above, two students did not post at all. You can explain this by looking to see if they are Sheryl and Tania. If they weren’t logging in, they won’t have been posting. Two other students did a bit better: one of them posted 9 times, and one posted 8 times.

By looking at the shape of the numbers you’ll soon see that most students are posting between 26 times and 79 times. However, more interesting than the safe majority are maybe the unsafe minority. The two who have not posted at all we have already noted. Six students have posted less than 20 times whereas most of the class have posted around 30 or 40 times. Equally concerning could be that far-outlier; the one student who has posted 159 times! Is he or she devoting enough time to other parts of the work, or are they starting to display obsessive tendencies?

In the real-world classroom, you may know Mister159 and you may have strategies for him. In the online space, it is too easy to forget that each of these avatars, these electronic tokens, represents a real human being with real needs. It can also be hard to learn those needs over the mediated channel of a computer network.

Brinkerhoff SCM

Robert Brinkerhoff’s Success Case Method uses impactful user stories to evaluate and later inform the design of educational interventions and programmes. To make a proper study of the method would be beyond the scope of this article, but I will give you the plain language guts of it here, and you can start using it straight away.

Here’s how:

Ignore the average and the median students
Focus on the outliers
- identify 1 – 3 students who did exceptionally well on the course
- identify 1 – 3 who did particularly poorly
Arrange to interview them
Document their stories.

Try to avoid surveying them with a set of standard questions. That practice is boring, and students may amuse themselves by skewing their answers, and it will miss some of the rich detail a conversation can uncover. Put those who did particularly poorly at their ease and explain that you are speaking with them in order to improve the course. Also, beyond the scope of this article is a primer on techniques of coaching, mentoring, and interpersonal communications that we will assume you have.

These are the stories we collected:

Mister159 had found the material easy and had self-appointed himself Helper of the Weak and Stupid (his phrase), a kind of unofficial assistant teacher. When we looked into it a bit more deeply by reading a sample of his posts, we found that he was sharing ready-made solutions to the problems!

The two students with 114 and 126 posts both really loved the subject. They collaborated on the work and were planning to study it in greater depth next year, continuing their collaboration. They were good friends out of school, too; high achievers.

At the other end of the scale, the orientation email had gone into Sheryl’s spam filter and Tania admitted she hadn’t read her email. We did a root-cause analysis on the latter to be sure we had not uncovered a case of cyberbullying. Happily, we hadn’t; she’d just not been focused.

The student who managed just eight posts said he found the subject uninteresting and he was thinking of changing his subjects.

As a result, the design changes the teacher might consider before the next delivery includes:

an optional section with some harder problems for those students who want extending
a once-a-week check on students who have not logged in
grading or peer upvoting of discussion posts that meet certain criteria.

Correlation is not causation, but…

That everyone who buys dried mangoes also buys sparkling water does not mean that the dried mangoes are causing them to be thirsty. From the shop keeper’s perspective, it doesn’t matter why, the fact is — they are. Therefore, he places a new line in sparkling spring water near the dried mangoes.

Between us, my wife and I have several decades experience as Moodle admins, and we have noticed that students who have silly Hotmail addresses very rarely complete. We don’t have causation here, only correlation. But, we do have a correlation, and it does work. Remember that list from the start of this article? Which student automatically goes on the at-risk list?

john.smith@stjosephs.school.nz
mary.roberts@stjosephs.school.nz
lbooth@middlebeach.school.nz
quackers@hotmail.com
garyb@chchpoly.ac.nz

This raises an issue of which we all need to be aware when we start profiling and working with student data to inform our course design and our interventions. Just because we’ve profiled quackers@hotmail.com does not give us the right to treat him or her differently to the rest of the cohort, that would be discrimination. But there is no law or moral code that says we can’t keep a watchful eye.

Designing for data collection

You can build a Moodle course with just two components: label, and forum. But, if you do take this streamlined, and dare I say lazy approach, you’re not going to be in a position to collect rich user data. The awesome power of Moodle is in all the activities it has built-in, and if you want to go down the plug-ins route, then hundreds more again.

Here are just a few ideas for how you can give your students interesting things to do in their learning environment.

Include the:

Choice activity to gather student opinion
Glossary activity to invite student contributions
Wiki activity to enable students to create exhibits
Blog activity so students can learn out loud
Journal activity for students to record their learning journeys.

How much more student data you can now collect to build a picture of their performance on and their engagement with the course.

There are some blocks you can usefully include in the sidebar design too.

Include the:

People block, so everyone can see who is engaging and who is at risk
Online users block to enable a real-time community of learners
Blog block to further develop the community of learners
Activity block to give easy access to everything there is for them to do.

The complete online tutor

I challenge you to change if you have not already done so. To be a great online teacher is not to create a whole mural of content and then stand back while the students admire it. To be a great online teacher is to create an interesting, safe, and supportive learning environment in which they can thrive. By measuring their progress, and encouraging them to make more, you will all have a wonderful experience. By knowing which students are at risk you can reach out to them and try and get them aboard your magic school bus.

CORE Blog

CORE Blog

Background