Impact Evaluation of Imagine
Math Facts in a Southwestern
Public School District
Kayla Freeman, Natasha Wilson, Drew Berrett
June 2024
Math Facts in a Southwestern
Public School District
Kayla Freeman, Natasha Wilson, Drew Berrett
June 2024
IntroductionFor students to be considered proficient in mathematics, they need to develop an understanding of
foundational mathematical concepts and be able to efficiently retrieve arithmetic facts (National
Mathematics Advisory Panel [NMAP], 2008). Students who lack proficiency and fluidity with their
math skills are more likely to experience mathematical deficiencies later in their education and
are less likely to develop the understanding needed for subsequent mathematical concepts
(Kilpatrick et al., 2001; NMAP, 2008; Rivera & Bryant, 1992; Shapiro, 2010; Woodward, 2006).
Research has indicated the importance of achieving math fact fluency particularly as it pertains
to later mathematics achievement (Geary, 2011). Specifically, this early math fact fluency is
critical for success in algebra and geometry (Nelson et al., 2016; Steel & Funnell, 2001).
Digital learning tools are a productive way for students to develop automaticity and fluency with
their mathematics skills (Lindeman, 2019). Imagine Math Facts by Imagine Learning is a digital
program that helps students gain fluency and automaticity in addition, subtraction, multiplication,
and division in an engaging 3D game environment. The program can be implemented in class or
as homework and provides students with a foundation for higher-order thinking to build pre-algebra
readiness. As such, students who utilize Imagine Math Facts as intended are expected to improve
and accelerate their mathematics skills proficiency.
In partnership with a southwestern school district, Imagine Learning conducted a study designed
to evaluate the efficacy of Imagine Math Facts. The primary research question was: how does
use of Imagine Math Facts impact elementary students’ mathematics achievement? Reported
study results demonstrate that the program positively impacted students’ mathematics proficiency
by comparing the performance of Imagine Math Facts students to a statistically similar group
of students who did not use the Imagine Math Facts program.
Methods
RESEARCH DESIGN
Imagine Learning partnered with a public school district in the southwestern United States
to evaluate how Imagine Math Facts impacted the math proficiency of its students. During
the 2022–2023 school year, Imagine Math Facts was made available to students in Grade 1
through Grade 5 and was used at teachers’ discretion. In many cases, it was implemented
in the classroom or at home if a teacher deemed it valuable to support the learning of an
individual student outside of the classroom.
This study was conducted retrospectively using data from the 2022–2023 school year to evaluate
the difference in mathematics achievement between treatment and control students. The
treatment group was comprised of all students who logged any usage in the Imagine Math
Facts program during the 2022–2023 school year, while the control group included all students
1Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
foundational mathematical concepts and be able to efficiently retrieve arithmetic facts (National
Mathematics Advisory Panel [NMAP], 2008). Students who lack proficiency and fluidity with their
math skills are more likely to experience mathematical deficiencies later in their education and
are less likely to develop the understanding needed for subsequent mathematical concepts
(Kilpatrick et al., 2001; NMAP, 2008; Rivera & Bryant, 1992; Shapiro, 2010; Woodward, 2006).
Research has indicated the importance of achieving math fact fluency particularly as it pertains
to later mathematics achievement (Geary, 2011). Specifically, this early math fact fluency is
critical for success in algebra and geometry (Nelson et al., 2016; Steel & Funnell, 2001).
Digital learning tools are a productive way for students to develop automaticity and fluency with
their mathematics skills (Lindeman, 2019). Imagine Math Facts by Imagine Learning is a digital
program that helps students gain fluency and automaticity in addition, subtraction, multiplication,
and division in an engaging 3D game environment. The program can be implemented in class or
as homework and provides students with a foundation for higher-order thinking to build pre-algebra
readiness. As such, students who utilize Imagine Math Facts as intended are expected to improve
and accelerate their mathematics skills proficiency.
In partnership with a southwestern school district, Imagine Learning conducted a study designed
to evaluate the efficacy of Imagine Math Facts. The primary research question was: how does
use of Imagine Math Facts impact elementary students’ mathematics achievement? Reported
study results demonstrate that the program positively impacted students’ mathematics proficiency
by comparing the performance of Imagine Math Facts students to a statistically similar group
of students who did not use the Imagine Math Facts program.
Methods
RESEARCH DESIGN
Imagine Learning partnered with a public school district in the southwestern United States
to evaluate how Imagine Math Facts impacted the math proficiency of its students. During
the 2022–2023 school year, Imagine Math Facts was made available to students in Grade 1
through Grade 5 and was used at teachers’ discretion. In many cases, it was implemented
in the classroom or at home if a teacher deemed it valuable to support the learning of an
individual student outside of the classroom.
This study was conducted retrospectively using data from the 2022–2023 school year to evaluate
the difference in mathematics achievement between treatment and control students. The
treatment group was comprised of all students who logged any usage in the Imagine Math
Facts program during the 2022–2023 school year, while the control group included all students
1Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
who did not. This study is a quasi-experimental design as assignment to the treatment and
control groups was not random. Statistical procedures were used to ensure baseline equivalence
of the treatment and control samples.
MEASURES
Multiple data sources were compiled to describe students, their performance, and their work
in Imagine Math Facts. Student mathematics proficiency outcomes were determined using
a standardized assessment. Student demographic data were collected to provide additional
information on student characteristics that may impact measures of learning outcomes. Data
from the Imagine Math Facts program were incorporated to evaluate student engagement in
Imagine Math Facts. These data sources are reviewed in more detail below.
Mathematics Proficiency. Students’ mathematics proficiency was determined using the
Renaissance Star (Ren Star) mathematics assessment for students in Grades 1 through 5.
The average number of days between the Fall 2022 and Spring 2023 assessments was 245
(242 days for students in the control group and 247 days for students in the treatment group).
Unified Scale scores from the Fall 2023 administration of the Ren Star math assessment were
used to establish baseline equivalence between study groups, and Spring 2023 Unified Scale
scores were used to estimate the effect of using Imagine Math Facts on mathematics proficiency.
The Ren Star Unified Scale score is a composite score on a Rasch score scale.
Student Demographics. Information on individual student demographic characteristics was also
collected. In particular, student grade level, ethnicity, gender, and special education status were
provided by the district and used to control for other factors that may impact student learning.
Imagine Math Facts Usage. Program usage data was obtained to determine students’ engagement
and progress in Imagine Math Facts. These data included the total minutes students spent in the
program, the number of math facts correctly answered, and whether each operation (addition,
subtraction, multiplication, division) was completed in the program.
ANALYTICAL SAMPLE
A total of 3,308 treatment students who used Imagine Math Facts and 2,791 control students
who did not use Imagine Math Facts were initially identified. To ensure that the baseline
characteristics of treatment and control students used in analyses were comparable, 1:1 nearest
neighbor propensity score matching without replacement was used to create a statistically
equivalent analytical sample.1 Control students were matched to treatment students based on
their Fall 2022 Ren Star Unified Scale score, gender, ethnicity, and special education status. This
matching process was completed on each grade individually before combining the matched
grade-level samples to create the two analytical samples: one with students in early elementary
grades (Grades 1 and 2) and a second with students in upper elementary grades (Grades 3
1 Propensity score matching was executed using the matchit function in R’s MatchIt package with the caliper set to .05 and
exact matching on grade, ethnicity, gender, and special education status.
2Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
control groups was not random. Statistical procedures were used to ensure baseline equivalence
of the treatment and control samples.
MEASURES
Multiple data sources were compiled to describe students, their performance, and their work
in Imagine Math Facts. Student mathematics proficiency outcomes were determined using
a standardized assessment. Student demographic data were collected to provide additional
information on student characteristics that may impact measures of learning outcomes. Data
from the Imagine Math Facts program were incorporated to evaluate student engagement in
Imagine Math Facts. These data sources are reviewed in more detail below.
Mathematics Proficiency. Students’ mathematics proficiency was determined using the
Renaissance Star (Ren Star) mathematics assessment for students in Grades 1 through 5.
The average number of days between the Fall 2022 and Spring 2023 assessments was 245
(242 days for students in the control group and 247 days for students in the treatment group).
Unified Scale scores from the Fall 2023 administration of the Ren Star math assessment were
used to establish baseline equivalence between study groups, and Spring 2023 Unified Scale
scores were used to estimate the effect of using Imagine Math Facts on mathematics proficiency.
The Ren Star Unified Scale score is a composite score on a Rasch score scale.
Student Demographics. Information on individual student demographic characteristics was also
collected. In particular, student grade level, ethnicity, gender, and special education status were
provided by the district and used to control for other factors that may impact student learning.
Imagine Math Facts Usage. Program usage data was obtained to determine students’ engagement
and progress in Imagine Math Facts. These data included the total minutes students spent in the
program, the number of math facts correctly answered, and whether each operation (addition,
subtraction, multiplication, division) was completed in the program.
ANALYTICAL SAMPLE
A total of 3,308 treatment students who used Imagine Math Facts and 2,791 control students
who did not use Imagine Math Facts were initially identified. To ensure that the baseline
characteristics of treatment and control students used in analyses were comparable, 1:1 nearest
neighbor propensity score matching without replacement was used to create a statistically
equivalent analytical sample.1 Control students were matched to treatment students based on
their Fall 2022 Ren Star Unified Scale score, gender, ethnicity, and special education status. This
matching process was completed on each grade individually before combining the matched
grade-level samples to create the two analytical samples: one with students in early elementary
grades (Grades 1 and 2) and a second with students in upper elementary grades (Grades 3
1 Propensity score matching was executed using the matchit function in R’s MatchIt package with the caliper set to .05 and
exact matching on grade, ethnicity, gender, and special education status.
2Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
through 5). Since some results might have been obscured were analyses conducted across all
grade levels together, outcomes were evaluated based on grade clusters and then on individual
grades. The resulting analytical samples included a total of 2,261 users of Imagine Math Facts and
2,261 non-users. Table 1 and Table 2 below describe the characteristics of the early elementary
and upper elementary analytical samples.
Table 1. Student Characteristics of the Early Elementary Analytical Sample
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 951 951
Average (SD) Fall 2022 Ren Star
Unified Scale Score 822.31 (62.81) 823.79 (58.73) .597 0.024
Grade >.999 <0.001
Grade 1 504 504
Grade 2 447 447
Gender >.999 <0.001
Female 484 484
Male 467 467
Ethnicity: Hispanic/Latino >.999 <0.001
No 509 509
Yes 442 442
Special Education >.999 <0.001
No 868 868
Yes 83 83
3Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
grade levels together, outcomes were evaluated based on grade clusters and then on individual
grades. The resulting analytical samples included a total of 2,261 users of Imagine Math Facts and
2,261 non-users. Table 1 and Table 2 below describe the characteristics of the early elementary
and upper elementary analytical samples.
Table 1. Student Characteristics of the Early Elementary Analytical Sample
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 951 951
Average (SD) Fall 2022 Ren Star
Unified Scale Score 822.31 (62.81) 823.79 (58.73) .597 0.024
Grade >.999 <0.001
Grade 1 504 504
Grade 2 447 447
Gender >.999 <0.001
Female 484 484
Male 467 467
Ethnicity: Hispanic/Latino >.999 <0.001
No 509 509
Yes 442 442
Special Education >.999 <0.001
No 868 868
Yes 83 83
3Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Table 2. Student Characteristics of the Upper Elementary Analytical Sample
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 1,310 1,310
Average (SD) Fall 2022 Ren Star
Unified Scale Score 982.60 (63.33) 980.77 (63.58) .460 0.029
Grade
Grade 3 408 408
Grade 4 457 457
Grade 5 445 445
Gender >.999 <0.001
Female 648 648
Male 662 662
Ethnicity: Hispanic/Latino >.999 <0.001
No 739 739
Yes 571 571
Special Education >.999 <0.001
No 1,184 1,184
Yes 126 126
ANALYTICAL APPROACH
Multiple linear regression was used to evaluate the differences in achievement between Imagine
Math Facts users and non-users in each analytical sample, controlling for baseline achievement
and demographic covariates. An indicator of whether a student was a control or treatment
student was included in the regressions as the primary predictor variable. Using multiple
linear regressions after propensity score matching ensured that any remaining differences in
the underlying treatment and control samples were controlled for by the regression model,
effectively isolating the impact of Imagine Math Facts.
4Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 1,310 1,310
Average (SD) Fall 2022 Ren Star
Unified Scale Score 982.60 (63.33) 980.77 (63.58) .460 0.029
Grade
Grade 3 408 408
Grade 4 457 457
Grade 5 445 445
Gender >.999 <0.001
Female 648 648
Male 662 662
Ethnicity: Hispanic/Latino >.999 <0.001
No 739 739
Yes 571 571
Special Education >.999 <0.001
No 1,184 1,184
Yes 126 126
ANALYTICAL APPROACH
Multiple linear regression was used to evaluate the differences in achievement between Imagine
Math Facts users and non-users in each analytical sample, controlling for baseline achievement
and demographic covariates. An indicator of whether a student was a control or treatment
student was included in the regressions as the primary predictor variable. Using multiple
linear regressions after propensity score matching ensured that any remaining differences in
the underlying treatment and control samples were controlled for by the regression model,
effectively isolating the impact of Imagine Math Facts.
4Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Results
IMAGINE MATH FACTS USAGE
Treatment students spent an average of 3.38 hours in Imagine Math Facts and passed an
average of 848.67 math facts. Average time in Imagine Math Facts varied by grade level,
with the highest average usage in Grade 3 and the lowest average usage in Grade 5. See
Figures 1 and 2 for a distribution of hours and facts lessons passed by grade.
Figure 1. Distribution of Hours Spent in Imagine Math Facts by Grade
14
12
8
2
53
6
2 41
0
10
4
Hours in Imagine Math Facts
Grade
Figure 2. Distribution of Facts Passed in Imagine Math Facts by Grade
Facts Passed in Imagine Math Facts
3500
3000
2500
250
53
2000
2 41
0
3250
2750
500
2250
1750
1250
750
1500
1000
Note: Outliers that fall above 1.5 times the interquartile range are not included in this figure to ensure
readability. The global maximum hours spent in Imagine Math Facts is 36.21 hours.
Grade
Note: Outliers that fall above 1.5 times the interquartile range are not included in this figure to ensure
readability. The global maximum number of facts passed in Imagine Math Facts is 9,510 facts.
5Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
IMAGINE MATH FACTS USAGE
Treatment students spent an average of 3.38 hours in Imagine Math Facts and passed an
average of 848.67 math facts. Average time in Imagine Math Facts varied by grade level,
with the highest average usage in Grade 3 and the lowest average usage in Grade 5. See
Figures 1 and 2 for a distribution of hours and facts lessons passed by grade.
Figure 1. Distribution of Hours Spent in Imagine Math Facts by Grade
14
12
8
2
53
6
2 41
0
10
4
Hours in Imagine Math Facts
Grade
Figure 2. Distribution of Facts Passed in Imagine Math Facts by Grade
Facts Passed in Imagine Math Facts
3500
3000
2500
250
53
2000
2 41
0
3250
2750
500
2250
1750
1250
750
1500
1000
Note: Outliers that fall above 1.5 times the interquartile range are not included in this figure to ensure
readability. The global maximum hours spent in Imagine Math Facts is 36.21 hours.
Grade
Note: Outliers that fall above 1.5 times the interquartile range are not included in this figure to ensure
readability. The global maximum number of facts passed in Imagine Math Facts is 9,510 facts.
5Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
2 The effect size is calculated using Hedges’ g computation following What Works Clearinghouse’s Procedures and Standards
Handbook, Version 5.0. The unadjusted standard deviations of the Spring 2023 scores can be found in Appendix A.
Treatment students in early elementary grades correctly answered more addition facts than
any other operation, whereas students in upper elementary grades correctly answered more
multiplication facts than any other problem (see Figure 3). Once students have mastered the
facts within an operation, as demonstrated by a passing score on a post-test, the operation is
considered completed in the Imagine Math Facts platform. In all grades, addition is the most
completed operation (see Figure 4).
Figure 3. Percent of Correct Facts in Imagine
Math Facts by Grade and Operation
100%
80%
60%
20%
5
40%
40% 21 3
Percent of Correct Facts
Grade
Figure 4. Percent of Students Who Completed
Each Operation by Grade
16%
8%
6%
2%
5
4%
40% 21 3
10%
12%
14%
Percent of Students Who Completed
Each Operation
Grade
Operation
● Addition ● Subtraction ● Multiplication ● Division
Operation
● Addition ● Subtraction ● Multiplication ● Division
PROGRAM IMPACT ON STUDENT ACHIEVEMENT
Use of Imagine Math Facts was found to generate a positive and statistically significant impact
on students’ mathematics proficiency for students in early elementary grades. Specifically, early
elementary students who used Imagine Math Facts scored an average of 6.71 points higher on
the Spring 2023 Ren Star mathematics assessment than otherwise similar non-user students,
B = 6.71, t(1895) = 3.879, p < .001. Program usage and the other covariates in the model accounted
for 65% of the variance found in Spring 2023 scores, R2 = .648, F(6,1895) = 582.3, p < .001. The
Hedges’ g effect size of Imagine Math Facts program usage is .106.2 Table 3 summarizes the
results of the multiple linear regression for students in early elementary grades. Use of Imagine
Math Facts was found to generate a non-significant impact on students’ mathematics proficiency
for students in upper elementary grades. Table 4 summarizes the results of the multiple linear
regression for students in upper elementary grades.
6Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Handbook, Version 5.0. The unadjusted standard deviations of the Spring 2023 scores can be found in Appendix A.
Treatment students in early elementary grades correctly answered more addition facts than
any other operation, whereas students in upper elementary grades correctly answered more
multiplication facts than any other problem (see Figure 3). Once students have mastered the
facts within an operation, as demonstrated by a passing score on a post-test, the operation is
considered completed in the Imagine Math Facts platform. In all grades, addition is the most
completed operation (see Figure 4).
Figure 3. Percent of Correct Facts in Imagine
Math Facts by Grade and Operation
100%
80%
60%
20%
5
40%
40% 21 3
Percent of Correct Facts
Grade
Figure 4. Percent of Students Who Completed
Each Operation by Grade
16%
8%
6%
2%
5
4%
40% 21 3
10%
12%
14%
Percent of Students Who Completed
Each Operation
Grade
Operation
● Addition ● Subtraction ● Multiplication ● Division
Operation
● Addition ● Subtraction ● Multiplication ● Division
PROGRAM IMPACT ON STUDENT ACHIEVEMENT
Use of Imagine Math Facts was found to generate a positive and statistically significant impact
on students’ mathematics proficiency for students in early elementary grades. Specifically, early
elementary students who used Imagine Math Facts scored an average of 6.71 points higher on
the Spring 2023 Ren Star mathematics assessment than otherwise similar non-user students,
B = 6.71, t(1895) = 3.879, p < .001. Program usage and the other covariates in the model accounted
for 65% of the variance found in Spring 2023 scores, R2 = .648, F(6,1895) = 582.3, p < .001. The
Hedges’ g effect size of Imagine Math Facts program usage is .106.2 Table 3 summarizes the
results of the multiple linear regression for students in early elementary grades. Use of Imagine
Math Facts was found to generate a non-significant impact on students’ mathematics proficiency
for students in upper elementary grades. Table 4 summarizes the results of the multiple linear
regression for students in upper elementary grades.
6Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Table 3. Overall Impact of Imagine Math Facts on Spring 2023 Ren Star Math Unified Scale Score in
Early Elementary Grades
Variables Estimate Standard Error p-value
Intercept 253.15 15.02 <.001
Imagine Math Facts User Indicator 6.71 1.73 <.001
Grade-Level Indicator
Grade 2 8.54 2.22 <.001
Fall 2022 Ren Star Unified Scale Score 0.77 0.02 <.001
Male Indicator 5.34 1.76 .002
Ethnicity: Hispanic/Latino Indicator -7.69 1.78 <.001
Special Education Indicator -11.08 3.13 <.001
Table 4. Overall Impact of Imagine Math Facts on Spring 2023 Ren Star Math Unified Scale Score in
Upper Elementary Grades
Variables Estimate Standard Error p-value
Intercept 260.21 15.44 <.001
Imagine Math Facts User Indicator -0.90 1.47 .540
Grade-Level Indicator
Grade 4 -2.28 2.05 .266
Grade 5 -10.08 2.44 <.001
Fall 2022 Ren Star Unified Scale Score 0.79 0.02 <.001
Male Indicator 1.58 1.51 .296
Ethnicity: Hispanic/Latino Indicator -4.50 1.52 .003
Special Education Indicator -15.93 2.71 <.001
7Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Early Elementary Grades
Variables Estimate Standard Error p-value
Intercept 253.15 15.02 <.001
Imagine Math Facts User Indicator 6.71 1.73 <.001
Grade-Level Indicator
Grade 2 8.54 2.22 <.001
Fall 2022 Ren Star Unified Scale Score 0.77 0.02 <.001
Male Indicator 5.34 1.76 .002
Ethnicity: Hispanic/Latino Indicator -7.69 1.78 <.001
Special Education Indicator -11.08 3.13 <.001
Table 4. Overall Impact of Imagine Math Facts on Spring 2023 Ren Star Math Unified Scale Score in
Upper Elementary Grades
Variables Estimate Standard Error p-value
Intercept 260.21 15.44 <.001
Imagine Math Facts User Indicator -0.90 1.47 .540
Grade-Level Indicator
Grade 4 -2.28 2.05 .266
Grade 5 -10.08 2.44 <.001
Fall 2022 Ren Star Unified Scale Score 0.79 0.02 <.001
Male Indicator 1.58 1.51 .296
Ethnicity: Hispanic/Latino Indicator -4.50 1.52 .003
Special Education Indicator -15.93 2.71 <.001
7Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Table 5. Impact of Imagine Math Facts on Spring 2023 Ren Star Math Unified Scale Scores by Grade
Grade
Estimate on Imagine Math
Facts Indicator Variable Standard Error p-value
Grade 1 -0.55 2.38 .816
Grade 2 15.52 2.48 <.001
Grade 3 -1.09 2.55 .669
Grade 4 -1.67 2.48 .499
Grade 5 0.02 2.59 .994
DIFFERENTIAL IMPACT BY GRADE
A series of analyses were further conducted to examine how the effects of Imagine Math Facts
varied across grade levels. Descriptive tables of unadjusted average Ren Star math Unified Scale
scores by grade can be found in Appendix A, and tables demonstrating baseline equivalence
by grade can be found in Appendix B. Grade 2 Imagine Math Facts users achieved statistically
significantly higher Spring 2023 Ren Star math Unified Scale scores than comparable non-users;
the difference is not statistically significant for students in other grades (Table 5). Complete
regression results can be found in Appendix C.
8Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Grade
Estimate on Imagine Math
Facts Indicator Variable Standard Error p-value
Grade 1 -0.55 2.38 .816
Grade 2 15.52 2.48 <.001
Grade 3 -1.09 2.55 .669
Grade 4 -1.67 2.48 .499
Grade 5 0.02 2.59 .994
DIFFERENTIAL IMPACT BY GRADE
A series of analyses were further conducted to examine how the effects of Imagine Math Facts
varied across grade levels. Descriptive tables of unadjusted average Ren Star math Unified Scale
scores by grade can be found in Appendix A, and tables demonstrating baseline equivalence
by grade can be found in Appendix B. Grade 2 Imagine Math Facts users achieved statistically
significantly higher Spring 2023 Ren Star math Unified Scale scores than comparable non-users;
the difference is not statistically significant for students in other grades (Table 5). Complete
regression results can be found in Appendix C.
8Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
ConclusionGiven the criticality of developing students’ mathematics skill fluency (NMAP, 2008), it is imperative
to understand the varying ways in which this can be best accomplished. Literature has pointed
to the benefits of digital learning programs (Lindeman, 2019) to accomplish this. The results
from this study corroborate and expand upon these findings.
Specifically, this study set out to examine the impact of Imagine Math Facts on the development
of mathematics achievement of students in Grades 1–5. Overall, findings revealed a statistically
significant impact on students’ Renaissance Star math scores for students in early elementary
grades. Early elementary students who used Imagine Math Facts scored 6.71 points higher on
the Spring 2023 administration of the Renaissance Star mathematics assessment than did
similar comparison students (p < .001).
A limitation of this study includes the relatively low average time spent in Imagine Math Facts
among the treatment group. Recommendations from Imagine Learning suggest that Imagine
Math Facts should be used as much as one hour per week throughout the school year, whereas
students in this sample used it a total of 3.38 hours on average through the entire school year.
Nevertheless, a statistically significant impact was found in lower elementary grade levels; this
could be due to the fact that addition is the most completed operation, and addition could
be most helpful in student growth on early elementary mathematics assessments focused
on basic numeracy whereas higher grades may be assessed on more complex mathematical
concepts and skills. More research is needed to determine the impact of Imagine Math Facts
on math proficiency in upper elementary grade levels.
In summary, this study provides evidence of effectiveness of Imagine Math Facts on mathematics
proficiency. Specifically, it demonstrates Imagine Math Facts’ impact on the achievement of
students in early elementary grades on the Renaissance Star mathematics assessment by
comparing the outcomes of students who participated in the program to those who did not.
9Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
to understand the varying ways in which this can be best accomplished. Literature has pointed
to the benefits of digital learning programs (Lindeman, 2019) to accomplish this. The results
from this study corroborate and expand upon these findings.
Specifically, this study set out to examine the impact of Imagine Math Facts on the development
of mathematics achievement of students in Grades 1–5. Overall, findings revealed a statistically
significant impact on students’ Renaissance Star math scores for students in early elementary
grades. Early elementary students who used Imagine Math Facts scored 6.71 points higher on
the Spring 2023 administration of the Renaissance Star mathematics assessment than did
similar comparison students (p < .001).
A limitation of this study includes the relatively low average time spent in Imagine Math Facts
among the treatment group. Recommendations from Imagine Learning suggest that Imagine
Math Facts should be used as much as one hour per week throughout the school year, whereas
students in this sample used it a total of 3.38 hours on average through the entire school year.
Nevertheless, a statistically significant impact was found in lower elementary grade levels; this
could be due to the fact that addition is the most completed operation, and addition could
be most helpful in student growth on early elementary mathematics assessments focused
on basic numeracy whereas higher grades may be assessed on more complex mathematical
concepts and skills. More research is needed to determine the impact of Imagine Math Facts
on math proficiency in upper elementary grade levels.
In summary, this study provides evidence of effectiveness of Imagine Math Facts on mathematics
proficiency. Specifically, it demonstrates Imagine Math Facts’ impact on the achievement of
students in early elementary grades on the Renaissance Star mathematics assessment by
comparing the outcomes of students who participated in the program to those who did not.
9Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
References
Geary, D. C. (2011). Cognitive predictors of achievement growth in mathematics: A 5-year longitudinal study.
Developmental Psychology, 47, 539–1552.
Kilpatrick, J., Swafford, J., & Findell, B. (Eds). (2001). Adding it up: Helping children learn mathematics. National
Academy Press.
Lindeman, A. (2019). Improving Math Fact Fluency in Elementary Students with Computer-assisted Instruction:
A Quasi-experimental Study - ProQuest.
https://www.proquest.com/docview/2322186312?pq-origsite=gscholar&fromopenview=true
National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National
Mathematics Advisory Panel. U.S. Department of Education.
Nelson, P. M., Parker, D. C., & Zaslofsky, A. F. (2016). The relative value of growth in math fact skills across late
elementary and middle school. Assessment for Effective Intervention, 41, 184–192.
Rivera, D. M., & Bryant, B. R. (1992). Mathematics instruction for students with special needs. Intervention in School
and Clinic, 28, 71–86.
Shapiro, E. S. (2010). Academic skills problems: Direct assessment and intervention (2nd ed.). Guildford Press.
Steel, S., & Funnell, E. (2001). Learning multiplication facts: A study of children taught by discovery methods in
England. Journal of Experimental Child Psychology, 79, 37–55.
What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0.
U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and
Regional Assistance (NCEE).
Woodward, J. (2006). Developing automaticity in basic multiplication facts: Integrating strategy instruction with time
practice drills. Learning Disability Quarterly, 29, 269–289.
10Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Geary, D. C. (2011). Cognitive predictors of achievement growth in mathematics: A 5-year longitudinal study.
Developmental Psychology, 47, 539–1552.
Kilpatrick, J., Swafford, J., & Findell, B. (Eds). (2001). Adding it up: Helping children learn mathematics. National
Academy Press.
Lindeman, A. (2019). Improving Math Fact Fluency in Elementary Students with Computer-assisted Instruction:
A Quasi-experimental Study - ProQuest.
https://www.proquest.com/docview/2322186312?pq-origsite=gscholar&fromopenview=true
National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National
Mathematics Advisory Panel. U.S. Department of Education.
Nelson, P. M., Parker, D. C., & Zaslofsky, A. F. (2016). The relative value of growth in math fact skills across late
elementary and middle school. Assessment for Effective Intervention, 41, 184–192.
Rivera, D. M., & Bryant, B. R. (1992). Mathematics instruction for students with special needs. Intervention in School
and Clinic, 28, 71–86.
Shapiro, E. S. (2010). Academic skills problems: Direct assessment and intervention (2nd ed.). Guildford Press.
Steel, S., & Funnell, E. (2001). Learning multiplication facts: A study of children taught by discovery methods in
England. Journal of Experimental Child Psychology, 79, 37–55.
What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0.
U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and
Regional Assistance (NCEE).
Woodward, J. (2006). Developing automaticity in basic multiplication facts: Integrating strategy instruction with time
practice drills. Learning Disability Quarterly, 29, 269–289.
10Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Appendix A
Table A1. Unadjusted Mean Ren Star Unified Scale Score by Grade Band
Fall 2022 (SD) Spring 2023 (SD) Mean Change
Grade 1
Imagine Math Facts (n = 504) 791.29 (45.89) 861.98 (50.39) 70.69
Comparison (n = 504) 784.96 (48.27) 857.97 (53.96) 73.01
Grade 2
Imagine Math Facts (n = 447) 860.43 (49.33) 931.99 (53.29) 71.56
Comparison (n = 447) 864.43 (49.20) 919.82 (59.23) 55.39
Grade 3
Imagine Math Facts (n = 408) 927.16 (49.70) 990.51 (50.47) 63.36
Comparison (n = 408) 929.50 (49.57) 993.26 (54.95) 63.76
Grade 4
Imagine Math Facts (n = 457) 985.19 (49.13) 1033.55 (57.20) 48.36
Comparison (n = 457) 986.83 (49.30) 1036.62 (58.65) 49.79
Grade 5
Imagine Math Facts (n = 445) 1025.39 (50.16) 1058.42 (57.02) 33.03
Comparison (n = 445) 1026.95 (49.96) 1059.68 (57.78) 32.73
Lower Elementary (Combined Grades 1–2)
Imagine Language & Literacy (n = 1,381) 823.79 (58.73) 894.89 (62.45) 71.10
Comparison (n = 1,381) 822.31 (62.81) 887.04 (64.36) 64.73
Upper Elementary (Combined Grades 3-5)
Imagine Math Facts (n = 1,310) 980.77 (63.58) 1028.60 (61.63) 47.82
Comparison (n = 1,310) 982.60 (63.33) 1030.95 (63.28) 48.34
11Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Table A1. Unadjusted Mean Ren Star Unified Scale Score by Grade Band
Fall 2022 (SD) Spring 2023 (SD) Mean Change
Grade 1
Imagine Math Facts (n = 504) 791.29 (45.89) 861.98 (50.39) 70.69
Comparison (n = 504) 784.96 (48.27) 857.97 (53.96) 73.01
Grade 2
Imagine Math Facts (n = 447) 860.43 (49.33) 931.99 (53.29) 71.56
Comparison (n = 447) 864.43 (49.20) 919.82 (59.23) 55.39
Grade 3
Imagine Math Facts (n = 408) 927.16 (49.70) 990.51 (50.47) 63.36
Comparison (n = 408) 929.50 (49.57) 993.26 (54.95) 63.76
Grade 4
Imagine Math Facts (n = 457) 985.19 (49.13) 1033.55 (57.20) 48.36
Comparison (n = 457) 986.83 (49.30) 1036.62 (58.65) 49.79
Grade 5
Imagine Math Facts (n = 445) 1025.39 (50.16) 1058.42 (57.02) 33.03
Comparison (n = 445) 1026.95 (49.96) 1059.68 (57.78) 32.73
Lower Elementary (Combined Grades 1–2)
Imagine Language & Literacy (n = 1,381) 823.79 (58.73) 894.89 (62.45) 71.10
Comparison (n = 1,381) 822.31 (62.81) 887.04 (64.36) 64.73
Upper Elementary (Combined Grades 3-5)
Imagine Math Facts (n = 1,310) 980.77 (63.58) 1028.60 (61.63) 47.82
Comparison (n = 1,310) 982.60 (63.33) 1030.95 (63.28) 48.34
11Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Appendix B
Table B1. Grade 1 Baseline Equivalence
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 504 504
Average (SD) Fall 2022 Ren Star
Unified Scale Score 784.96 (48.27) 791.29 (45.89) .033 0.134
Gender >.999 <0.001
Female 253 253
Male 251 251
Ethnicity: Hispanic/Latino >.999 <0.001
No 251 251
Yes 253 253
Special Education >.999 <0.001
No 465 465
Yes 39 39
Table B2. Grade 2 Baseline Equivalence
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 447 447
Average (SD) Fall 2022 Ren Star
Unified Scale Score 864.43 (49.20) 860.43 (49.33) .225 0.081
Gender >.999 <0.001
Female 231 231
Male 216 216
Ethnicity: Hispanic/Latino >.999 <0.001
No 258 258
Yes 189 189
Special Education >.999 <0.001
No 403 403
Yes 44 44
12Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Table B1. Grade 1 Baseline Equivalence
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 504 504
Average (SD) Fall 2022 Ren Star
Unified Scale Score 784.96 (48.27) 791.29 (45.89) .033 0.134
Gender >.999 <0.001
Female 253 253
Male 251 251
Ethnicity: Hispanic/Latino >.999 <0.001
No 251 251
Yes 253 253
Special Education >.999 <0.001
No 465 465
Yes 39 39
Table B2. Grade 2 Baseline Equivalence
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 447 447
Average (SD) Fall 2022 Ren Star
Unified Scale Score 864.43 (49.20) 860.43 (49.33) .225 0.081
Gender >.999 <0.001
Female 231 231
Male 216 216
Ethnicity: Hispanic/Latino >.999 <0.001
No 258 258
Yes 189 189
Special Education >.999 <0.001
No 403 403
Yes 44 44
12Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Table B3. Grade 3 Baseline Equivalence
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 408 408
Average (SD) Fall 2022 Ren Star
Unified Scale Score 929.50 (49.57) 927.16 (49.70) .499 0.047
Gender >.999 <0.001
Female 216 216
Male 192 192
Ethnicity: Hispanic/Latino >.999 <0.001
No 249 249
Yes 159 159
Special Education >.999 <0.001
No 368 368
Yes 40 40
Table B4. Grade 4 Baseline Equivalence
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 457 457
Average (SD) Fall 2022 Ren Star
Unified Scale Score 986.83 (49.30) 985.19 (49.13) .615 0.033
Gender >.999 <0.001
Female 209 209
Male 248 248
Ethnicity: Hispanic/Latino >.999 <0.001
No 254 254
Yes 203 203
Special Education >.999 <0.001
No 411 411
Yes 46 46
13Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 408 408
Average (SD) Fall 2022 Ren Star
Unified Scale Score 929.50 (49.57) 927.16 (49.70) .499 0.047
Gender >.999 <0.001
Female 216 216
Male 192 192
Ethnicity: Hispanic/Latino >.999 <0.001
No 249 249
Yes 159 159
Special Education >.999 <0.001
No 368 368
Yes 40 40
Table B4. Grade 4 Baseline Equivalence
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 457 457
Average (SD) Fall 2022 Ren Star
Unified Scale Score 986.83 (49.30) 985.19 (49.13) .615 0.033
Gender >.999 <0.001
Female 209 209
Male 248 248
Ethnicity: Hispanic/Latino >.999 <0.001
No 254 254
Yes 203 203
Special Education >.999 <0.001
No 411 411
Yes 46 46
13Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Table B5. Grade 5 Baseline Equivalence
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 445 445
Average (SD) Fall 2022 Ren Star
Unified Scale Score 1026.95 (49.96) 1025.39 (50.16) .642 0.031
Gender >.999 <0.001
Female 223 223
Male 222 222
Ethnicity: Hispanic/Latino >.999 <0.001
No 236 236
Yes 209 209
Special Education >.999 <0.001
No 405 405
Yes 40 40
14Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Group
Comparison
Students
Imagine
Math Facts
Students p-value
Standardized
Mean Difference
(SMD)
n 445 445
Average (SD) Fall 2022 Ren Star
Unified Scale Score 1026.95 (49.96) 1025.39 (50.16) .642 0.031
Gender >.999 <0.001
Female 223 223
Male 222 222
Ethnicity: Hispanic/Latino >.999 <0.001
No 236 236
Yes 209 209
Special Education >.999 <0.001
No 405 405
Yes 40 40
14Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Table C1. Grade 1 Regression Results
Variables Estimate Standard Error p-value
Intercept 294.12 20.67 <.001
Imagine Math Facts User Indicator -0.55 2.38 .816
Fall 2022 Ren Star Unified Scale Score 0.72 0.03 <.001
Male Indicator 8.17 2.41 .001
Ethnicity: Hispanic/Latino Indicator -9.80 2.43 <.001
Special Education Indicator -13.01 4.52 .004
Appendix C
Table C2. Grade 2 Regression Results
Variables Estimate Standard Error p-value
Intercept 198.20 23.62 <.001
Imagine Math Facts User Indicator 15.52 2.48 <.001
Fall 2022 Ren Star Unified Scale Score 0.84 0.03 <.001
Male Indicator 1.48 2.54 .559
Ethnicity: Hispanic/Latino Indicator -4.72 2.59 .070
Special Education Indicator -7.55 4.30 .080
Table C3. Grade 3 Regression Results
Variables Estimate Standard Error p-value
Intercept 336.24 26.56 <.001
Imagine Math Facts User Indicator -1.09 2.55 .669
Fall 2022 Ren Star Unified Scale Score 0.71 0.03 <.001
Male Indicator 6.16 2.64 .020
Ethnicity: Hispanic/Latino Indicator -0.88 2.69 .744
Special Education Indicator -23.65 4.62 <.001
15Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Variables Estimate Standard Error p-value
Intercept 294.12 20.67 <.001
Imagine Math Facts User Indicator -0.55 2.38 .816
Fall 2022 Ren Star Unified Scale Score 0.72 0.03 <.001
Male Indicator 8.17 2.41 .001
Ethnicity: Hispanic/Latino Indicator -9.80 2.43 <.001
Special Education Indicator -13.01 4.52 .004
Appendix C
Table C2. Grade 2 Regression Results
Variables Estimate Standard Error p-value
Intercept 198.20 23.62 <.001
Imagine Math Facts User Indicator 15.52 2.48 <.001
Fall 2022 Ren Star Unified Scale Score 0.84 0.03 <.001
Male Indicator 1.48 2.54 .559
Ethnicity: Hispanic/Latino Indicator -4.72 2.59 .070
Special Education Indicator -7.55 4.30 .080
Table C3. Grade 3 Regression Results
Variables Estimate Standard Error p-value
Intercept 336.24 26.56 <.001
Imagine Math Facts User Indicator -1.09 2.55 .669
Fall 2022 Ren Star Unified Scale Score 0.71 0.03 <.001
Male Indicator 6.16 2.64 .020
Ethnicity: Hispanic/Latino Indicator -0.88 2.69 .744
Special Education Indicator -23.65 4.62 <.001
15Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Table C4. Grade 4 Regression Results
Variables Estimate Standard Error p-value
Intercept 201.41 28.03 <.001
Imagine Math Facts User Indicator -1.67 2.48 .499
Fall 2022 Ren Star Unified Scale Score 0.85 0.03 <.001
Male Indicator 2.30 2.55 .366
Ethnicity: Hispanic/Latino Indicator -4.76 2.55 .062
Special Education Indicator -14.77 4.46 .001
Table C5. Grade 5 Regression Results
Variables Estimate Standard Error p-value
Intercept 225.80 29.70 <.001
Imagine Math Facts User Indicator 0.02 2.59 .994
Fall 2022 Ren Star Unified Scale Score 0.82 0.03 <.001
Male Indicator -3.36 2.64 .203
Ethnicity: Hispanic/Latino Indicator -7.65 2.62 .004
Special Education Indicator -8.88 4.97 .074
16Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
Variables Estimate Standard Error p-value
Intercept 201.41 28.03 <.001
Imagine Math Facts User Indicator -1.67 2.48 .499
Fall 2022 Ren Star Unified Scale Score 0.85 0.03 <.001
Male Indicator 2.30 2.55 .366
Ethnicity: Hispanic/Latino Indicator -4.76 2.55 .062
Special Education Indicator -14.77 4.46 .001
Table C5. Grade 5 Regression Results
Variables Estimate Standard Error p-value
Intercept 225.80 29.70 <.001
Imagine Math Facts User Indicator 0.02 2.59 .994
Fall 2022 Ren Star Unified Scale Score 0.82 0.03 <.001
Male Indicator -3.36 2.64 .203
Ethnicity: Hispanic/Latino Indicator -7.65 2.62 .004
Special Education Indicator -8.88 4.97 .074
16Impact Evaluation of Imagine Math Facts in a Southwestern Public School District
imaginelearning.com/math-facts
877-725-4257 • solutions@imaginelearning.com 1380999626 2407
877-725-4257 • solutions@imaginelearning.com 1380999626 2407