How Do We Know If a Math Program Works?

When a school adopts a new math program, a common question is: Does it work? This chapter reviews the pieces needed to assess an educational program and read an impact study. It describes three main ideas:

• It is not possible to show whether an educational program works in the abstract. The question we can ask is: “What difference does using this program make compared with the available alternative, for a specific outcome, population, and context?”
• Not all evidence carries the same weight or is equally informative. The rigor of an impact study depends on its capacity to estimate what would have happened to those same students without the program, and this requires constructing a fair comparison with equivalent students who did not receive it.
• A good impact evaluation study also requires a sound theory of change: what should change, through what mechanism, and under what conditions? It is important to define what is expected to happen when the program is used, what should change, and to understand implementation as a fundamental part of the process.

The chapter concludes with eight questions for reading any study and the most common questions from those receiving an effectiveness report.