Evaluation of online and blended learning
Version 1.0, October 2014
Evaluation is central to ensuring the quality of teaching and learning, and academic practice.
Tom Reeves & Hedberg (2003) propose the Conducting Evaluations approach for e-learning:
Step 1: Identify decisions that must be made about e-learning.
Why evaluate? Identify the reasons for an evaluation.
Is to adopt, expand, improve, abandon, or review?
Step 2: Clarify questions that must be addressed to guide decisions
What do you want to learn? Establish the goals of an evaluation.
For example: Who is enrolled in e-learning and why? How can LMS be improved? What is the impact on access, on performance? Did students achieve higher order learning outcomes?
Step 3: Select methods
Who will you ask? Determine your sample.
What will you ask? Create your instruments.
Observations, interviews, focus groups, questionnaires, data log analysis, expert review, usability studies.
Don't ask: Which test should we use? but What counts as evidence that learning has occurred?
Step 4: Collect the data
Triangulate (mixed methods), revise data collection methods – accept limitations but aim for quality.
Step 5: Report findings so they influence decisions in time
Evaluate alignment: Outcomes, learner tasks, assessment, technological features.
Report early and often, use multiple formats, engage stakeholders in focus groups.
don't make decisions, people do.
Don't reduce complexity by oversimplifying, describe e-learning in many ways.
Planning an evaluation
In planning an evaluation, establish the basic purpose of the evaluation, for example, formative or summative, then the focus in the evaluation context, and finally the strategies for gathering evidence, e.g., focus group, observation, survey, and so on.
Will involve ascertaining the match between the 'learning and instructional goals' and the design specification.
Peer-review of the design model/architecture (combined with interview assisted think aloud).
Will involve ascertaining whether the educational innovation matches predefined design criteria. This involves a small group of students as well as the tutors.
Direct observation (combined with structured responses via checklist).
Focus group or individual interview (semi-structured).
Involves gathering data from a larger sample of users in the first full-scale use of the materials in a real teaching and learning situation.
Direct observation (combined with structured responses via checklist/questionnaire).
Analysis of user interactions and products of their learning activities.
User's self-reporting which includes post-hoc comments gained through querying, and analysis of Critical Reflections on the learning activities.
Semi-structured and open-ended questionnaires of user satisfaction with the materials.
Outcome evaluation, effects on specifically defined learning outcomes.
Impact evaluation will examine the overall or net effects of the program or technology as a whole.
Outcome evaluation can be achieved through looking at student performance on the assigned tasks.
Impact evaluation can be achieved through the use of questionnaires, and focus group debriefs.
Monitoring and on-going evaluation
Secondary analysis re-examines existing data to address new questions.
Integration of innovation. The extent to which the study materials are forming an integral part of teaching and learning of the subject.
Time on task. This is an estimation of time spent by teachers and students on the required tasks.
Secondary analysis is achieved through critical analysis of existing data.
Integration of innovation. Can be ascertained by examining current practice.
Time on task. Can be ascertained by reviewing logs kept by teachers and students.
- Evaluating e-learning developments, Learning and Development Centre, University of Warwick, UK.
- Harvey, J. (1998). Evaluation cookbook.
Reeves, T. & Hedberg, J. (2003). Interactive Learning Systems Evaluation. , Englewood Cliffs: Educational Technology Publications
Reeves, T. (2011). Using Evaluation Tools Effectively. Educause.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.