Forum Higher Ed & Training Forum: Educational Research and Learning Design: The No Significant Difference problem

By: James Dalziel

[ Reply | Forward ]

As with my prior post in this community, I've been writing a chapter for a new book with the Carnegie Foundation and MIT on Learning Design and LAMS, and had to edit out a whole section on Educational Research and Learning Design due to word limits. So I've posted it below for those who are interested in this topic. I need to write this issue up properly some day, but hope this is useful in the meantime.


Implications of Learning Design for Educational Research

Learning Design may have special benefits for educational research. As Learning Designs provide a detailed articulation of the learning process, they may provide a more solid foundation for experimental/comparison-based educational research. One explanation for the lack of significant differences in comparisons of educational innovations with control groups is that there are so many small differences in the learning experiences between student groups, that a myriad of confounding factors wash away any potentially measurable difference arising from the innovation.

The introduction of Learning Design to these comparisons could provided a much closer comparison of learning experiences between groups, with some of the confounding factors removed or mitigated as a result of a greater similarity in educational contexts (apart from the relevant difference under investigation). Using Learning Design in this way provides no guarantee that significant differences would be found, but it may help to remove some of the “noise” in comparisons of student groups in educational research, and hence provide a better chance of measuring any genuine differences that are being masked by confounding factors.

A second benefit of Learning Design for educational research arises from contexts where a rich record of student behaviour is recorded as a result of running the design. This is most common when Learning Designs are implemented online using a system that automatically records student contributions (such as LAMS), but there is no theoretical reason why similar student records couldn’t be achieved in a face-to-face context through audio/video recording and collation of written notes. In either case, capturing a rich record of student behaviour as they progress through a Learning Design may provide educational researchers with new insights into the strengths and weaknesses of different educational approaches, as well as the potential to compare student learning processes and outcomes between online and face-to-face Learning Designs.

It is worth noting that richer records of student behaviour within Learning Designs may lead to a wider variety of measurable outcomes for research. There is a tendency in some educational research to focus only on measures such as test scores on multiple choice questions; whereas in certain contexts, this may be a poor measure of whether any changes in student learning have occurred. Rich records from learning designs would permit other kinds of quantitative measures (such as the number or length of forum postings) as well as potentially more meaningful qualitative analyses such as changes in a student’s written comments about a concept over the life of a Learning Design (or a series of designs). As this last point suggests, detection of some differences may require comparisons of each student over time (ie, repeated measures), rather than simple comparison of test score averages between groups at the end of a study.

My personal suspicion, for what it is worth, is that there are real differences to be found in certain contexts, but we need studies that carefully use Learning Design to reduce the “noise” in group comparisons, as well as new kinds of measures to detect differences (many of which may be qualitative, and based on repeated measures experimental approaches). I also suspect we need longer timeframes to detect differences in many contexts – a comparison of groups after a single lesson may not be sufficient – we may need a series of controlled comparisons over many sessions (and hence multiple Learning Designs) before differences are readily apparent.

Why could the impact of Learning Design on educational research matter for open education? Governments, charitable foundations and others have invested significant funding in the development of open educational content. The “no significant difference” literature of educational research suggests that in comparisons of online educational resources with traditional teaching methods, there is no reliable difference in measured educational outcomes. This may not be a concern for those projects that have a primary goal of bringing educational resources to those that would not otherwise have access to them (such as OpenCourseware). But many projects have a focus on improving existing education through the additional use of open educational resources, and to date we have little credible evidence that this has been achieved. Improvements in educational research arising from the use of Learning Design may help us to identify those contexts where genuine differences can be found (and replicated), and hence which topics, which teaching methods and which kinds of technology are deserving of greater attention.

Posted by James Dalziel