Evaluation of online and blended learning

Version 1.0, October 2014

Evaluation is central to ensuring the quality of teaching and learning, and academic practice.

Tom Reeves & Hedberg (2003) propose the Conducting Evaluations approach for e-learning:

Step 1: Identify decisions that must be made about e-learning.

Why evaluate? Identify the reasons for an evaluation.

Is to adopt, expand, improve, abandon, or review?

Step 2: Clarify questions that must be addressed to guide decisions

What do you want to learn? Establish the goals of an evaluation.

For example: Who is enrolled in e-learning and why? How can LMS be improved? What is the impact on access, on performance? Did students achieve higher order learning outcomes?

Step 3: Select methods

Who will you ask? Determine your sample.
What will you ask? Create your instruments.

Observations, interviews, focus groups, questionnaires, data log analysis, expert review, usability studies.
Don't ask: Which test should we use? but What counts as evidence that learning has occurred?

Step 4: Collect the data

Triangulate (mixed methods), revise data collection methods – accept limitations but aim for quality.

Step 5: Report findings so they influence decisions in time

Evaluate alignment: Outcomes, learner tasks, assessment, technological features.
Report early and often, use multiple formats, engage stakeholders in focus groups.

Data don't make decisions, people do.
Don't reduce complexity by oversimplifying, describe e-learning in many ways.


Planning an evaluation

In planning an evaluation, establish the basic purpose of the evaluation, for example, formative or summative, then the focus in the evaluation context, and finally the strategies for gathering evidence, e.g., focus group, observation, survey, and so on.

Type

Focus

Strategies

Formative evaluation:

Design evaluation

 

Will involve ascertaining the match between the 'learning and instructional goals' and the design specification.

Peer-review of the design model/architecture (combined with interview assisted think aloud).

Prototype evaluation

 

Will involve ascertaining whether the educational innovation matches predefined design criteria. This involves a small group of students as well as the tutors.

Direct observation (combined with structured responses via checklist).

Focus group or individual interview (semi-structured).

Implementation evaluation

 

Involves gathering data from a larger sample of users in the first full-scale use of the materials in a real teaching and learning situation.

Direct observation (combined with structured responses via checklist/questionnaire).

Analysis of user interactions and products of their learning activities.

User's self-reporting which includes post-hoc comments gained through querying, and analysis of Critical Reflections on the learning activities.

Semi-structured and open-ended questionnaires of user satisfaction with the materials.

Summative evaluation

Outcome evaluation, effects on specifically defined learning outcomes.

Impact evaluation will examine the overall or net effects of the program or technology as a whole.

Outcome evaluation can be achieved through looking at student performance on the assigned tasks.

Impact evaluation can be achieved through the use of questionnaires, and focus group debriefs.

Monitoring and on-going evaluation

Secondary analysis re-examines existing data to address new questions.

Integration of innovation. The extent to which the study materials are forming an integral part of teaching and learning of the subject.

Time on task. This is an estimation of time spent by teachers and students on the required tasks.

Secondary analysis is achieved through critical analysis of existing data.

Integration of innovation. Can be ascertained by examining current practice.

Time on task. Can be ascertained by reviewing logs kept by teachers and students.

Evaluation resources

References

Reeves, T. & Hedberg, J. (2003). Interactive Learning Systems Evaluation. , Englewood Cliffs: Educational Technology Publications

Reeves, T. (2011). Using Evaluation Tools Effectively. Educause.


This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Return to Resource Library