Measuring learner engagement in online education

I recently presented a paper at #amee_els about how myself and colleagues at the Univeristy of Otago have gone about developing a questionnaire to measure learner engagement in online medical education.

It seems to have struck a chord, and a number of people from the conference have asked if they can use this approach in their work.

The answer is yes, of course, and the slides from the conference are posted below. A journal article describing the development is being written, and I will post here when that is published. For now, however, I want to point out a couple of things:

  • the version of the instrument referred to in the slides is essentially a beta version
  • we’re currently looking at the psychometric properties, and some adjustment is necesary
  • I suspect that some of the items are redundant and/or too specific to a learning context and/or not only related to engagement.

What I am working on now is a different version of the instrument that is more easily generalised. I’m currently testing this out in a project and will be looking at the pyschometrics of this approach soon. Updates will be posted here.

How you use this approach

If you wish to use this approach, go ahead. All I ask is:

  1. Let me know by emailing steve.gallagher [at] Tell me a bit about how you’re using it and any other feedback you wish.
  2. Please use this attribution if you write anything about this work


Gallagher, S. (2015). Measuring learning engagement in online medical education. Paper presented at the AMEE eLearning Symposium, “Shaping the Future of Technology-Enhanced Learning”, September 5-6, 2015, Glasgow, UK. DOI: 10.13140/RG.2.1.2873.7767

Here’s a link to a Google Doc version of the survey:

AMEE and more London

This is a brief update only, I need more time to put together my thoughts on the last 10 days or so, but I wanted to get a few key points down.

Glasgow was amazing, despite walking out of Central Station and immediately feeling like you’d walked into a pub just before a fight is about to break out. A block or so away from that and you feel more relaxed. The AMEE eLearning Symposium was a great experience, including keynotes from Diana Laurillard and Stephen Downes – both of which were thought provoking and fascinating. A number of excellent presentations and an interesting debate between Natalie Lafferty and David Cook about the reality versus hype of eLearning kept the brain ticking over. I met some excellent people on twitter and in person, and particularly enjoyed hearing about student co-design in action from Dundee and Glasgow Universities. It might be my new favourite thing.

The full AMEE2015 conference during the next three days was also stimulating. The first keynote on mindfulness and mind-body medicine included a guided visualisation for approximately 2,500 people finished with us biting an imaginary lemon – it was incredibly well done and convincingly laid out a physiological explanation for the benefits of mindful meditation. Other excellent keynotes included a fascinating description of current thinking about emotion and cognition, was great to see some cognitive science at a medical conference.

Back in London, I’ve visited Imperial College Faculty of Medicine to learn about their eLearning projects and use of iPads for clinical students. Much to ponder here and a good relationship building exercise for me. I also spent a day with my ex-colleagues at City University, London and learnt about their learning spaces and Moodle projects.

More on this will come later, including pulling together multiple tweets from the conference – it’s probably the most active I’ve been on social media at a conference and was a very productive networking tool.

Having an amazing time, will post some photos later.


I’m here, and loving being back in London.

A few notes re international travel I was reminded about:

  • LAX sucks
  • You won’t be able to work on your conference paper in the airport or on the plane. Don’t even try
  • Never have the eggs

But London is awesome. Heathrow was painless, the express and tube to Euston was a piece of cake (ah the smell of the tube…). My one night in TheWesley Hotel was great, it’s a brilliant location if you enjoy Bloomsbury or are travelling out of Euston or Kings Cross.

I walked off my jet lag via the lovely streets around NW1 and Bloomsbury, and visited the Wellcome Institute. Amazing place, with some incredible medical exhibits.

That thing…

…where you realise you haven’t blogged for almost 2 years. Well, it’s been a busy time!

  • I grew a beard.
Cheesey beardy Steve

Cheesey beardy Steve

  • We went to Australia and I presented (twice) at ANZAHPE 2014
  • We’ve built a house.
House in progress

House in progress

I now have a full-time role as Lecturer in eLearning at the Dunedin School of Medicine. The great thing about this, apart from working in an area that I love, is that I get more time to do research. I’m going to be presenting on some work that I started last year with colleagues in the Otago Medical School at #amee-els in Glasgow, and visiting e-learning and medical education types in London, Montréal, and finishing up attending medicineX|ed at Stanford. So September 2015 is going to busy, but I’m going to try and blog as I go.

Flipping things


A while ago I said I was going to start topic pages on this blog, and add the top 5 things related to that topic in one, easy-to-find place.Well, I’m still working on that idea.

But in the meantime, I’ve been looking at ways of harvesting things of interest. I was an early user of, which still has a pile of things I harvested from a number of years ago, but I found i rarely went back to it. I used to use Bloglines to harvest RSS feeds, and keep the good ones in delicious, but my workflow fell apart after a while. More recently I used Google Reader for RSS, and have since moved to Feedly. But the problem remains about where to keep “the good stuff”.

Enter Flipboard magazines. I’ve used Flipboard for scanning RSS and other feeds of interest, but only recently started creating what Flipboard calls magazines as a place to curate content. Adding content is straightforward via a bookmarklet. Now, if I see something in Feedly, or follow a link in Twitter, I can easily capture it to Flipboard.

This is an experiment and may not continue, but for now – two big topics in education in recent years now have a magazine each where I am keeping things of note. Check them out here:

Competence in organizational learning

Competence or competency management in learning and development is a popular approach to measuring learning needs and allocating learning or training opportunities. It fits in with the philosophy that learning and development are designed to move someone from their current level of competence to the organizations desired competence level for a particular role.
It’s something I’ve seen put in place in corporate environments, and I’ve helped designed competency frameworks and implement systems to measure and allocate learning to fill competency gaps. As I did more of this is a corporate context I became less confident that it was useful to an individual learner, or than large organizations really had the buy in to make them work.
Anyway, I’ve just read something in a 2008 book from Stephen Downes called The Future of Online Learning: 10 years on. He put it like this (my emphasis):

The idea of competences is that they are based on identifiable skills or capacities, and hence are not rooted in a body of content but rather in a student’s personal growth. (Karampiperis, Demetrios, & Demetrios, 2006) As such, students are able to select their own track or achievement path through a competence domain, as informed by their own interests, employer needs, or in the case of younger students, parental guidance. Each competence, meanwhile, corresponds to a selection of learning resources (and specifically, learning objects). (de-Marcos, Pages, Martinez, & Gutierrez, 2007)
It is not clear that such a system will meet the needs of learners. Insofar as this is a form of autonomous learning, it is not clear that it supports collaboration or cooperation. Moreover, it is not clear that an outcomes driven system is what students require; many valuable skills and aptitudes – art appreciation, for example – are not identifiable as an outcome. This becomes evident when we consider how learning is to be measured. In traditional learning, success is achieved not merely by passing the test but in some way being recognized as having achieved expertise. A test-only system is a coarse system of measurement for a complex achievement.
Downes, S. (2008). The future of online learning: Ten years on. pp 17-18

As usual, Downes says it better than I ever could.