Latent Class Evaluation in Educational Trials: What Percentage of Children Benefits from an Intervention?

Abstract

Educational stakeholders are keen to know the magnitude and importance of different interventions. However, the way evidence is communicated to support understanding of the effectiveness of an intervention is controversial. Typically studies in education have used the standardised mean difference as a measure of the impact of interventions. This measure, commonly known as the effect size, is problematic, in terms of how it is interpreted and understood. In this study, we propose a "gain index" as an alternative metric for quantifying and communicating the effectiveness of an intervention. This is estimated as the difference in the percentage of children who make positive gains between the intervention and control groups. Analysis of four randomized controlled trials in education supports the expectation that most children make progress due to normal school activities, which is independent of the intervention. This study elaborates a method to illustrate how trials with a positive gain index and with a higher percentage of pupils with positive gain in the intervention group can be used to communicate which trials are effective in improving educational outcomes.