In my last post, I unpacked the Learning Management System (LMS) and the Learning Record Store (LRS). In it, I introduced the idea of capturing fine-grained student data using the Experience Application Programming Interface (xAPI) to inform programme, course, and module design. I felt it may have left many readers with a bit of a tall order, and in need of a simpler place to start.
That was the case for me back in 2010 at a regional polytechnic. With classes of 30 students, lectures to prepare, assignments to mark, at the same time developing an online community and resource, to also try to go deep into user data analysis would have been a bridge too far.
In this post, I plan to offer a starting place; some low-hanging fruit. It’s a complete starter pack for someone who has a course in Moodle and no idea where to start. I am sure it would easily transfer to any mainstream LMS.
Identifying students at risk
In your role as an online teacher (tutor, instructor, facilitator), the first thing you will want to do is identify those students who are at risk.
There is no greater risk for a student than not being present. In a physical classroom that is obvious; there are some empty seats. On the first day of the class, Sheryl and Tania are missing. The teacher wonders if they are aware they should be in this class. Perhaps they did not read the notice that said the room had changed. Perhaps they have discovered a clash with another class. After the class, the teacher has a duty to follow up and find out where these two girls are.
The same thing happens in the online world, but all too often the absence goes unnoticed. The teacher’s attention is focused on the people who are in, say, the introductory webinar — whose sound is not working, who have too many distracting questions. Especially if the enrolment is large, Sheryl and Tania may not be missed.
There are ways to resolve the problem, like recruiting an assistant facilitator, but that’s not our focus here; I was using it only by way of illustration. Let’s focus on Sheryl and Tania and use the tools available to us to determine if they are at risk.
By clicking on the link in the People block, Moodle displays not only a list of the people in the course but also when they last logged on. If it says “NEVER” beside Sheryl and Tania, then you know you have two students at risk.
If that was all an online teacher ever did — follow up on the NEVERs — that would be good in itself. But, if you have an ounce of geek in you, then there is a whole lot more that you can do.
Exploratory data analysis
The thing about exploratory data analysis is… it’s exploratory. Think of your data as being like a territory, and you set out on foot to get to know it.
Whether you are a beginner, a journeyman, or an expert in data analysis you will always start by just considering the data. Look at it, shuffle it about a bit, make a cup of tea, ponder. Don’t rush in, you may miss something that is staring you in the face.
For example, look at these five email addresses:
I won’t say anything for now, but we’ll come back to this list later in the post.
John Tukey is the father of exploratory data analysis, and he wrote a textbook of that title. Despite all the advances since he wrote it in 1977, it is still a good place to start your journey. If you are the textbook type, that is. You may prefer to simply learn by doing.
Let’s do a little Tukey-type thing now. He talks about, ”scratching down numbers”, and this method is called stem-and-leaf. It gives you a quick visualisation and, therefore, the beginnings of an understanding of the numbers.
I’ll make that a little easier to see:
Let’s say that you’d like to quantify a feeling you have that your cohort of 60 students is contributing quite well to the discussion forum. The stem to the left of the line is the tens column; to the right the ones. As you can see in the image above, two students did not post at all. You can explain this by looking to see if they are Sheryl and Tania. If they weren’t logging in, they won’t have been posting. Two other students did a bit better: one of them posted 9 times, and one posted 8 times.
By looking at the shape of the numbers you’ll soon see that most students are posting between 26 times and 79 times. However, more interesting than the safe majority are maybe the unsafe minority. The two who have not posted at all we have already noted. Six students have posted less than 20 times whereas most of the class have posted around 30 or 40 times. Equally concerning could be that far-outlier; the one student who has posted 159 times! Is he or she devoting enough time to other parts of the work, or are they starting to display obsessive tendencies?
In the real-world classroom, you may know Mister159 and you may have strategies for him. In the online space, it is too easy to forget that each of these avatars, these electronic tokens, represents a real human being with real needs. It can also be hard to learn those needs over the mediated channel of a computer network.
Robert Brinkerhoff’s Success Case Method uses impactful user stories to evaluate and later inform the design of educational interventions and programmes. To make a proper study of the method would be beyond the scope of this article, but I will give you the plain language guts of it here, and you can start using it straight away.
- Ignore the average and the median students
- Focus on the outliers
- identify 1 – 3 students who did exceptionally well on the course
- identify 1 – 3 who did particularly poorly
- Arrange to interview them
- Document their stories.
Try to avoid surveying them with a set of standard questions. That practice is boring, and students may amuse themselves by skewing their answers, and it will miss some of the rich detail a conversation can uncover. Put those who did particularly poorly at their ease and explain that you are speaking with them in order to improve the course. Also, beyond the scope of this article is a primer on techniques of coaching, mentoring, and interpersonal communications that we will assume you have.
These are the stories we collected:
Mister159 had found the material easy and had self-appointed himself Helper of the Weak and Stupid (his phrase), a kind of unofficial assistant teacher. When we looked into it a bit more deeply by reading a sample of his posts, we found that he was sharing ready-made solutions to the problems!
The two students with 114 and 126 posts both really loved the subject. They collaborated on the work and were planning to study it in greater depth next year, continuing their collaboration. They were good friends out of school, too; high achievers.
At the other end of the scale, the orientation email had gone into Sheryl’s spam filter and Tania admitted she hadn’t read her email. We did a root-cause analysis on the latter to be sure we had not uncovered a case of cyberbullying. Happily, we hadn’t; she’d just not been focused.
The student who managed just eight posts said he found the subject uninteresting and he was thinking of changing his subjects.
As a result, the design changes the teacher might consider before the next delivery includes:
- an optional section with some harder problems for those students who want extending
- a once-a-week check on students who have not logged in
- grading or peer upvoting of discussion posts that meet certain criteria.
Correlation is not causation, but…
That everyone who buys dried mangoes also buys sparkling water does not mean that the dried mangoes are causing them to be thirsty. From the shop keeper’s perspective, it doesn’t matter why, the fact is — they are. Therefore, he places a new line in sparkling spring water near the dried mangoes.
Between us, my wife and I have several decades experience as Moodle admins, and we have noticed that students who have silly Hotmail addresses very rarely complete. We don’t have causation here, only correlation. But, we do have a correlation, and it does work. Remember that list from the start of this article? Which student automatically goes on the at-risk list?
This raises an issue of which we all need to be aware when we start profiling and working with student data to inform our course design and our interventions. Just because we’ve profiled email@example.com does not give us the right to treat him or her differently to the rest of the cohort, that would be discrimination. But there is no law or moral code that says we can’t keep a watchful eye.
Designing for data collection
You can build a Moodle course with just two components: label, and forum. But, if you do take this streamlined, and dare I say lazy approach, you’re not going to be in a position to collect rich user data. The awesome power of Moodle is in all the activities it has built-in, and if you want to go down the plug-ins route, then hundreds more again.
Here are just a few ideas for how you can give your students interesting things to do in their learning environment.
- Choice activity to gather student opinion
- Glossary activity to invite student contributions
- Wiki activity to enable students to create exhibits
- Blog activity so students can learn out loud
- Journal activity for students to record their learning journeys.
How much more student data you can now collect to build a picture of their performance on and their engagement with the course.
There are some blocks you can usefully include in the sidebar design too.
- People block, so everyone can see who is engaging and who is at risk
- Online users block to enable a real-time community of learners
- Blog block to further develop the community of learners
- Activity block to give easy access to everything there is for them to do.
The complete online tutor
I challenge you to change if you have not already done so. To be a great online teacher is not to create a whole mural of content and then stand back while the students admire it. To be a great online teacher is to create an interesting, safe, and supportive learning environment in which they can thrive. By measuring their progress, and encouraging them to make more, you will all have a wonderful experience. By knowing which students are at risk you can reach out to them and try and get them aboard your magic school bus.
“The Success Case Method deliberately looks at the most, and least, successful participants of a program. The purpose is not to examine the average performance – rather, by identifying and examining the extreme cases…” Success Case Method at betterevaluation.org
If you are geeking out, then you might like to look at the Moodle database schema. You will soon see what you are and are not going to be able to harvest from what Moodle stores.
On the face of it, this TEDx talk by Ben Wellington is nothing to do with education. But it is a great, and amusing, lesson on how to make data useful and engaging through storytelling — in any field of endeavour. Recommended.
Follow the series
This is the second in a series of posts on online learning design. Here are the others in this series so far: