Technology is making its way into many classrooms across the country, easing the way for schools to use educational software as one part of instruction. While the proliferation of ed-tech products and software programs can seem unending, the research backing up the claims made by companies has yet to catch up. This environment makes it challenging for school leaders to make informed decisions about products that have a big impact on how teachers teach and students learn.
A study completed by the independent, not-for-profit research and development agency WestEd is an effort by the MIND Research Institute to document if its program, ST Math, makes a measurable difference on student test scores. The institute commissioned the study to compare the state test scores of 19,980 second-to-fifth-graders in California, who had access to ST Math’s program in the 2010-2011 school year, with a comparison group of students who did not have the program. All teachers who used ST Math got the same training and teacher support from the ST Math staff on how to use the software. They also received help on how to integrate the technology for use in face-to-face teaching.
Researchers found that 65 percent of students who had access to ST Math scored proficient or advanced on the California Standards Test, as compared with 62.5 percent in the comparison group. That difference is statistically significant, but in real terms doesn’t appear to be a very big effect.
However, not all classrooms that used ST Math took a comprehensive approach. For grades that “fully implemented the program,” meaning at least 85 percent of the students used the program to cover at least 50 percent of the content for the class, the difference was greater. These schools invested time in program adoption, rather than just having access to the program and using it intermittently. For that group, 68 percent of students scored proficient.
“The biggest limitation is the fact that we’re doing a matched comparison study,” said John Rice, senior research associate at WestEd. It’s a quasi-experimental research setup -- not the gold standard randomized control study by any means -- but the best that could be done in this case. Researchers adjusted for factors like race, socioeconomics and starting levels of proficiency, to make the data as even as possible, but comparing grade-level cohorts in this manner is inherently opaque.