Can a math program really make a difference in learning?

What do we know about the impact of math materials and programs on learning? The accumulated research since the beginning of this century offers a general conclusion: curriculum materials can improve outcomes, but their effects are usually moderate and depend largely on how they are implemented. Moreover, we still know much less than desirable about which programs work, for whom, and under what conditions. A review of the available evidence yields the following conclusions:

• Rigorous evidence remains scarce. Most of it comes from the United States, and even there, many widely used programs have never been evaluated using designs capable of convincingly estimating their impact. 
• Obtaining causal evidence is costly and difficult. A rigorous evaluation can require several years, thousands of students, numerous schools, considerable funding, and centers willing to accept that program assignment is determined randomly. 
• When improvements are detected, they are usually moderate, but they can be relevant on a large scale. Meta-analyses place the average advantage of replacing some curriculum materials with others at around +0.05 to +0.10 standard deviations (SD). Among the studies reviewed by the What Works Clearinghouse and Evidence for ESSA, the average is +0.13 SD. These are modest effects, but the comparison is not between receiving or not receiving instruction: the control group continues to use its regular materials. Therefore, these figures represent the additional improvement attributable to replacing one alternative with another. If both have similar costs, even a moderate improvement can be important when it reaches all students. 
• The same program can produce different results depending on where, how, and against what it is evaluated. The observed impact depends on the quality of the alternative used by the comparison group, the test used to measure learning, the time elapsed since the intervention, and the quality of implementation. Evaluations closely aligned with the program content tend to find larger effects than external tests, and some of the improvements may fade over time. 
• Materials work best when accompanied by practice-linked teacher training. Studies synthesized by Pellegrini et al. (2021) and Lynch et al. (2019) find higher effect sizes when the intervention combines curriculum materials with coaching and professional development (the effect size grows between +0.08 and +0.10 SD). Evidence suggests that delivering good materials is not enough: helping teachers use them well increases their impact.

• The balance between cost and benefit. Intensive interventions such as tutoring or small group work achieve, on average, high effects. Something similar happens with coaching. However, these interventions require far more human resources and are difficult to scale. Curriculum materials accompanied by training produce smaller effects per student, but they can reach practically all classrooms at a relatively low additional cost.