of 24

Current View
JUNE 2023
Impact Evaluation of
Imagine Math® in a
Charter School NetworkKayla Freeman, Macey Cartwright, and Drew Berrett
IntroductionProficiency in early mathematics plays an important role in students’ future academic success
and career opportunities (Claessens & Engel, 2013; National Research Council, 2012; Shanley et
al., 2017). Imagine Math by Imagine Learning offers a digital supplemental mathematics learning
solution designed to support Pre-Kindergarten through Geometry students of all backgrounds.
The program provides adaptive and developmentally-appropriate instruction that focuses on
building students’ conceptual understanding of mathematics. To supplement and enhance
digital lessons, on-demand instruction by live, certified teachers is available to deliver scaffolded
instruction and ensure that students receive timely assistance as needed.
This study aimed to evaluate the efficacy of Imagine Math by addressing the research
question: how does participation in Imagine Math impact student achievement in mathematics?
To accomplish this, Imagine Learning partnered with a large, southeastern charter school
network which implemented Imagine Math across multiple schools with the intent to improve
student math performance. Reported study results demonstrate how this program impacted
students’ NWEA MAP Growth Math performance by comparing the performance of Imagine
Math students to a highly similar group of students who did not use Imagine Math.
MethodsPOPULATION
Imagine Learning partnered with a charter school network in the southeastern region of the
United States to evaluate how Imagine Math had impacted the success of its students. During
the 2020–2021 school year, Imagine Math was made available to students in Kindergarten
through Grade 8 across 16 schools in Florida and North Carolina. In these schools, Imagine
Math was used at teachers’ discretion. In many cases, it was used in the classroom or at home
if a teacher deemed it valuable to support the learning of an individual student. A total of 9,015
students in those schools used the program. Conversely, there were 2,311 students in those
schools who did not use Imagine Math. In addition, data were collected for 18,532 students
from 35 other schools in the charter network in Florida and North Carolina where Imagine
Math was not offered to students. In total, data were collected for 9,015 treatment students
who used Imagine Math and 20,843 control students who did not use Imagine Math.
RESEARCH DESIGN
This study was conducted retrospectively using data from the 2020–2021 school year. It evaluated
the difference in mathematics achievement between treatment and control students. The
treatment group was comprised of all students who logged any usage in the Imagine Math
program during the 2020–2021 school year, while the control group included all students who
did not. Assignment to the treatment and control groups was not random, so this study is a
quasi-experimental design, and statistical procedures were used to ensure baseline equivalence
2Impact Evaluation of Imagine Math in a Charter School Network
of the treatment and control samples. Because use of Imagine Math in the schools was
determined for individual students (rather than for entire classrooms or schools), statistical
corrections for clustering were not required.
MEASURES
Multiple data sources were compiled to describe students, their performance, and their work
in Imagine Math. Student math proficiency outcomes were determined using a standardized
progress-monitoring assessment. Student demographic data were collected to provide additional
information on student characteristics that may impact measures of learning outcomes. Data
from the Imagine Math program were incorporated to evaluate student engagement in Imagine
Math. These data sources are reviewed in more detail below.
Math Proficiency. Students’ math proficiency was determined using the NWEA MAP Growth
Math assessment. NWEA MAP Growth Math RIT scores were obtained for students in Fall 2020
and Spring 2021. The average number of days between the Fall 2020 and Spring 2021 assessments
was 228 (226 days for students in the control group and 230 days for students in the treatment
group). Fall 2020 scores were used to establish baseline equivalence between study groups,
and Spring 2021 scores were used to estimate the effect of Imagine Math on math proficiency.
Student Demographics. Information was collected on individual student demographic
characteristics including grade level, gender, and race/ethnicity. Since the 2020–2021 school
year was impacted by the COVID-19 pandemic, information on where students completed
their NWEA MAP Growth Math assessment (in-person, remote, or hybrid) was also collected.
Imagine Math Usage. Program usage data was also obtained to determine students’
engagement and progress in Imagine Math. These data included the total minutes students
spent in the program and the number of lessons students passed (defined as lessons that
students completed and achieved at least 75% accuracy on the post-lesson quiz).
ANALYTICAL SAMPLE
To ensure that the baseline characteristics of treatment and control students used in analyses
were comparable, 1:1 nearest neighbor propensity score matching without replacement was
used to create a statistically equivalent analytical sample.1 Control students were matched to
treatment students based on their Fall 2020 NWEA Map Growth Math RIT score, testing location2,
and all demographic information available: gender and race/ethnicity. This matching process
was completed on each grade individually before combining the matched grade level samples
to create the total analytical sample. The resulting analytical sample included 9,015 users of
Imagine Math and 9,015 non-users. Table 1 below describes the characteristics of the sample.
To ensure that the results of the analyses were not sensitive to the final analytic sample chosen,
a second analytic sample was made that required exact matches on all characteristics listed
above. Further details of this sample and the resulting analyses can be found in Appendix A.
1 Propensity score matching was executed using the matchit function in R’s MatchIt package.
2 Note that this study was completed during the 2020–2021 school year, which was impacted during the COVID-19 pandemic.
This study therefore considers student’s testing location (in-person, flexible, mobile, or not reported) as a covariate in analyses.
3Impact Evaluation of Imagine Math in a Charter School Network
Table 1. Student Characteristics of the Analytical Sample
ANALYTICAL APPROACH
Multiple linear regression was used to evaluate the differences in Spring 2021 NWEA MAP
Growth achievement between Imagine Math users and non-users, controlling for Fall 2021
MAP Growth achievement and other covariates (including grade level, gender, race/ethnicity,
and testing location). An indicator of whether a student was a control or treatment student
was included in the regression as the primary predictor variable. Using multiple linear regressions
after propensity score matching ensured that any remaining differences in the underlying
treatment and control samples were controlled for by the regression model, effectively isolating
the impact of Imagine Math.
Group Subgroup
Comparison
Students
Imagine Math
Students p-value
Standardized
Mean Difference
(SMD)
n 9,015 9,015
Average (SD) Fall
2020 NWEA MAP
Growth Math RIT
Score
197.56 (27.63) 197.76 (27.70) .619 0.007
Grade Level
Kindergarten 924 924
>.999 <0.001
Grade 1 926 926
Grade 2 968 968
Grade 3 1,244 1,244
Grade 4 1,261 1,261
Grade 5 1,166 1,166
Grade 6 970 970
Grade 7 900 900
Grade 8 656 656
Gender Female 4,688 4,637 .456 0.011Male 4,327 4,378
Race/Ethnicity
White
(Non-Hispanic)
4,230 4,077
.013 0.057
Asian 401 458
Black 1,835 1,821
Hispanic 2,061 2,085
Multi-Ethnic 424 488
Other 64 86
Testing Location
In-Person 5,767 5,541
.007

0.052

Flexible 113 123
Mobile 2,282 2,434
Not Selected 853 917
4Impact Evaluation of Imagine Math in a Charter School Network
ResultsIMAGINE MATH USAGE
Treatment students spent an average of 17.6 hours in Imagine Math and passed an average of
21.2 lessons. Average time in Imagine Math varied by grade level, with the highest average usage
in Grade 3 and the lowest average usage in Grade 2. See Figures 1 and 2 for a distribution of
hours and lessons passed by grade.
Figure 1. Distribution of Hours Spent in Imagine Math by Grade
Note: Outliers that fall above 1.5 times the
interquartile range are not included in this
figure to ensure readability. The global
maximum hours spent in Imagine Math is
193.1 hours.
0
50
60
40
20
Grade
70
10
30
K 1 2 3 4 5 6 7 8
Hours in Imagine Math

Figure 2. Distribution of Lessons Passed in Imagine Math by Grade
Note: Outliers that fall above 1.5 times the
interquartile range are not included in this
figure to ensure readability. The global
maximum number of lessons passed in
Imagine Math is 265.0 lessons.
0
50
80
40
20
Grade
90
10
30
K 1 2 3 4 5 6 7 8
Lessons Passed in Imagine Math
100
70
60

5Impact Evaluation of Imagine Math in a Charter School Network
PROGRAM IMPACT ON STUDENT ACHIEVEMENT
Overall, use of Imagine Math was found to generate a positive and statistically significant
impact on students’ mathematics performance. Specifically, students who used Imagine Math
scored an average of 2.00 points higher on the Spring 2021 NWEA MAP Growth Math assessment
than otherwise similar non-user students, B = 2.00, t(18010) = 13.40, p < .001. Program usage
and the other covariates in the model accounted for 85% of the variance found in Spring
2021 scores, R2 = .847, F(19,18010) = 5,242, p < .001. The Hedges’ g effect size of Imagine Math
program usage is .078.3 Table 2 summarizes the results of the multiple linear regression. The
covariate-adjusted mean Spring 2021 score was 209.09 for Imagine Math users and 207.08
for non-users.
Table 2. Overall Impact of Imagine Math on Spring 2021 MAP Growth Math RIT Scores
Coefficient Estimate Standard Error p-value
Intercept 52.79 0.92 <.001
Imagine Math User
Indicator
2.00 0.15 <.001
Grade-Level
Indicator
1 1.08 0.35 .002
2 1.56 0.38 <.001
3 4.89 0.41 <.001
4 6.18 0.45 <.001
5 6.53 0.50 <.001
6 5.66 0.52 <.001
7 7.02 0.55 <.001
8 10.42 0.59 <.001
Fall 2020 MAP
Growth RIT Score
0.76 0.01 <.001
Male Indicator 0.80 0.15 <.001
Race/Ethnicity
Indicator
Asian 2.50 0.37 <.001
Black -2.91 0.20 <.001
Hispanic -1.04 0.20 <.001
Multi-Ethnic -0.17 0.35 .621
Other -1.04 0.83 .210
Testing Location Flexible -1.31 0.66 .048
Mobile -1.61 0.18 <.001
Not Selected -0.85 0.26 <.001
3 The effect size is calculated using Hedges’ g computation following What Works Clearinghouse’s Procedures and Standards
Handbook, Version 5.0. The unadjusted standard deviations of the Spring 2021 scores can be found in Appendix B.
6Impact Evaluation of Imagine Math in a Charter School Network
DIFFERENTIAL IMPACT BY GRADE LEVEL
A series of analyses were further conducted to examine whether the effects of Imagine Math
varied across grade level. Descriptive tables of unadjusted average NWEA MAP Growth Math
RIT scores by grade can be found in Appendix B, and tables demonstrating baseline equivalence
by grade can be found in Appendix C. Imagine Math users had statistically significantly higher
Spring 2021 NWEA MAP Growth RIT scores than comparable non-users for Grade 1 through
Grade 8 students. Multiple linear regressions revealed positive coefficients that were statistically
significant for all grades except Kindergarten, where the p-value of .066 approached statistical
significance (Table 3). Complete regression results can be found in Appendix D.
Table 3. Impact of Imagine Math on Spring 2021 MAP Growth Math RIT Scores by Grade Level
ConclusionThis study provides ESSA Tier 3 evidence of the efficacy of Imagine Math on student math
achievement for students in Grades K–8 by comparing students who participated in Imagine
Math with those who did not during the 2020–2021 school year. Results show that students
who participated in Imagine Math scored two points higher on the Spring 2021 administration
of the NWEA MAP Growth Math test than did similar comparison students. This difference
was statistically significant. Similarly, results by grade level show that users scored between
one and four points higher than comparable non-users, and the differences were statistically
significant for Grades 1 through 8. Thus, this study provides evidence that the use of Imagine
Math supports students’ mathematics achievement.
Grade Level
Estimate on
Imagine Math
Indicator Variable Standard Error p-value
Kindergarten 0.93 0.51 .066
Grade 1 1.88 0.52 <.001
Grade 2 2.59 0.44 <.001
Grade 3 1.77 0.36 <.001
Grade 4 1.84 0.35 <.001
Grade 5 1.36 0.35 <.001
Grade 6 2.04 0.39 <.001
Grade 7 1.94 0.46 <.001
Grade 8 4.59 0.69 <.001
7Impact Evaluation of Imagine Math in a Charter School Network
References
Claessens, A., & Engel, M. (2013). How Important is Where you Start? Early Mathematics
Knowledge and Later School Success. Teachers College Record, 115(6), 1–29.
National Research Council. (2012). Education for life and work: Developing transferable
knowledge and skills in the 21st century. Committee on Defining Deeper Learning and
21st Century Skills, James W. Pellegrino and Margaret L. Hilton (Eds.), Board on Testing and
Assessment and Board on Science Education, Division of Behavioral and Social Sciences and
Education. The National Academies Press.
Shanley, L., Clarke, B., Doabler, C. T., Kurtz-Nelson, E., & Fien, H. (2017). Early number skills gains
and mathematics achievement: Intervening to establish successful early mathematics
trajectories. The Journal of Special Education, 51(3), 177–188.
What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards
handbook, version 5.0. U.S. Department of Education, Institute of Education Sciences,
National Center for Education Evaluation and Regional Assistance (NCEE).
8Impact Evaluation of Imagine Math in a Charter School Network
Appendix A
To ensure that observed results were not sensitive to the matching process used to select the
analytical sample, a second analytical sample was created using a different, more restrictive
procedure. Rather than using propensity score matching, treatment students were matched
to control students if they exactly matched on Fall 2020 NWEA MAP Growth Math RIT score,
grade level, gender, race, and testing location. With this more restrictive matching procedure,
matches could not be identified for some treatment students. These treatment students were
dropped from the analytical sample. This process resulted in a sample with 6,234 treatment
(Imagine Math user) students and 6,234 control (non-user) students; Table A1 demonstrates
the equivalence of the samples.
Table A1. Baseline Equivalence of Exact Matched Sample
Group Subgroup
Comparison
Students
Imagine Math
Students p-value
Standardized
Mean Difference
(SMD)
n 6,234 6,234
Average (SD) Fall
2020 NWEA MAP
Growth Math RIT
Score
197.04 (26.89) 197.04 (26.89) >.999 <0.001
Grade Level
Kindergarten 607 607
>.999 <0.001
Grade 1 626 626
Grade 2 682 682
Grade 3 842 842
Grade 4 861 861
Grade 5 779 779
Grade 6 731 731
Grade 7 628 628
Grade 8 478 478
Gender Female 3,249 3,249 >.999 <0.001Male 2,985 2,985
Race/Ethnicity
White
(Non-Hispanic)
2,978 2,978
>.999 <0.001
Asian 112 112
Black 1,446 1,446
Hispanic 1,532 1,532
Multi-Ethnic 141 141
Other 25 25
Testing Location
In-Person 4,423 4,423
>.999 <0.001Flexible 22 22
Mobile 1,508 1,508
Not Selected 281 281
9Impact Evaluation of Imagine Math in a Charter School Network
The results of the analyses are largely the same as the analytical sample from the original
matching procedure. Students who used Imagine Math scored statistically significantly (M = 1.93
points) higher on the Spring 2021 NWEA MAP Growth Math assessment than students who did
not use Imagine Math, B = 1.93, t(12448) = 11.63, p < .001. Program usage and the other covariates
in the model accounted for 86% of the variance found in Spring 2021 scores, R2 = .861, F(19,12448)
= 4054, p < .001. The effect size of Imagine Math program usage is .078.4 The covariate-adjusted
mean Spring 2021 score was 208.74 for Imagine Math users and 206.81 for non-users.
When investigating the results by grade, Imagine Math users in all grade levels scored higher
on the Spring 2021 NWEA MAP Growth Math assessment, and that difference was statistically
significant in all grades except Grade 1. See Table A2 below for a summary of the coefficients
of the Imagine Math indicator variables that indicate the expected difference in performance
between students who used Imagine Math and students who did not use Imagine Math based
on multiple linear regressions that control for the same covariates as the main analysis.
Table A2. Summary of Regression Results for Exact Matched Sample by Grade and Overall
Grade Level
Estimate on
Imagine Math
Indicator Variable Standard Error p-value
Kindergarten 1.57 0.56 .005
Grade 1 1.00 0.58 .088
Grade 2 2.66 0.47 <.001
Grade 3 1.65 0.39 <.001
Grade 4 1.66 0.38 <.001
Grade 5 1.37 0.41 <.001
Grade 6 2.02 0.45 <.001
Grade 7 1.67 0.54 .002
Grade 8 4.65 0.77 <.001
All Grades Combined 1.93 0.17 <.001
4 The effect size is calculated using Hedges’ g computation following What Works Clearinghouse’s Procedures and Standards
Handbook, Version 5.0. The unadjusted standard deviation of Spring 2021 NWEA MAP Growth Math RIT scores for treatment
students was 24.95 and for control students was 24.62.
10Impact Evaluation of Imagine Math in a Charter School Network
Appendix B
Table B1. Unadjusted Mean MAP Growth Math RIT Scores by Grade Appendix B
Fall 2020 (SD) Spring 2021 (SD) Mean Change
Grade K Imagine Math (n = 924) 148.30 (13.10) 166.42 (12.98) 18.12
Comparison (n = 924) 148.43 (13.44) 165.49 (13.20) 17.05
Grade 1 Imagine Math (n = 926) 168.24 (12.93) 183.04 (13.13) 14.80
Comparison (n = 926) 168.14 (13.82) 181.13 (14.47) 12.99
Grade 2 Imagine Math (n = 968) 181.87 (13.13) 194.21 (11.97) 12.35
Comparison (n = 968) 182.16 (13.64) 191.84 (12.75) 9.68
Grade 3 Imagine Math (n = 1244) 194.12 (12.29) 206.52 (11.86) 12.40
Comparison (n = 1244) 193.62 (12.21) 204.59 (11.98) 10.97
Grade 4 Imagine Math (n = 1261) 205.52 (11.87) 216.52 (13.38) 11.00
Comparison (n = 1261) 205.10 (11.43) 214.34 (12.67) 9.24
Grade 5 Imagine Math (n = 1166) 214.84 (12.60) 223.63 (14.83) 8.79
Comparison (n = 1166) 214.56 (12.81) 222.00 (14.99) 7.44
Grade 6 Imagine Math (n = 970) 218.12 (12.51) 225.60 (14.60) 7.48
Comparison (n = 970) 217.91 (12.65) 223.37 (14.59) 5.48
Grade 7 Imagine Math (n = 900) 224.28 (14.79) 231.49 (17.27) 7.25
Comparison (n = 900) 223.96 (14.51) 229.25 (16.51) 5.29
Grade 8 Imagine Math (n = 656) 227.73 (16.19) 238.92 (20.17) 11.20
Comparison (n = 656) 227.45 (15.96) 234.12 (19.29) 6.66
All Grades
Combined
Imagine Math (n = 9,015) 197.76 (27.70) 209.19 (25.80) 11.43
Comparison (n = 9,015) 197.56 (27.63) 207.05 (25.39) 9.49
11Impact Evaluation of Imagine Math in a Charter School Network
Appendix C
Table C1. Kindergarten Baseline Equivalence
Group Subgroup
Comparison
Students
Imagine Math
Students p-value SMD
n 924 924
Average (SD) Fall
2020 NWEA MAP
Growth Math RIT
Score
148.43 (13.44) 148.30 (13.10) .832 0.010
Gender Female 487 479 .744 0.017Male 437 445
Race/Ethnicity
White
(Non-Hispanic)
287 286
.992 0.033
Asian 52 59
Black 158 158
Hispanic 365 359
Multi-Ethnic 61 61
Other 1 1
Testing Location
In-Person 652 645
.797 0.047Flexible 4 4
Mobile 159 174
Not Selected 109 101
12Impact Evaluation of Imagine Math in a Charter School Network
Group Subgroup
Comparison
Students
Imagine Math
Students p-value SMD
n 968 968
Average (SD) Fall
2020 NWEA MAP
Growth Math RIT
Score
182.16 (13.64) 181.87 (13.13) .627 0.022
Gender Female 487 483 .892 0.008Male 481 485
Race/Ethnicity
White
(Non-Hispanic)
513 484
.512 0.094
Asian 48 55
Black 233 227
Hispanic 135 150
Multi-Ethnic 33 42
Other 6 10
Testing Location
In-Person 608 561
.175 0.101Flexible 4 5
Mobile 258 286
Not Selected 98 116
Table C3. Grade 2 Baseline Equivalence
Group Subgroup
Comparison
Students
Imagine Math
Students p-value SMD
n 926 926
Average (SD) Fall
2020 NWEA MAP
Growth Math RIT
Score
168.14 (13.82) 168.24 (12.93) .872 0.008
Gender Female 482 478 .889 0.009Male 444 448
Race/Ethnicity
White
(Non-Hispanic)
338 327
.996 0.028
Asian 44 43
Black 142 142
Hispanic 333 343
Multi-Ethnic 64 66
Other 5 5
Testing Location
In-Person 609 595
.915 0.033Flexible 5 5
Mobile 210 222
Not Selected 102 104
Table C2. Grade 1 Baseline Equivalence
13Impact Evaluation of Imagine Math in a Charter School Network
Table C5. Grade 4 Baseline Equivalence
Group Subgroup
Comparison
Students
Imagine Math
Students p-value SMD
n 1,261 1,261
Average (SD) Fall
2020 NWEA MAP
Growth Math RIT
Score
205.10 (11.43) 205.52 (11.87) .362 0.036
Gender Female 657 632 .339 0.040Male 604 629
Race/Ethnicity
White
(Non-Hispanic)
585 581
.953 0.042
Asian 68 76
Black 273 274
Hispanic 265 253
Multi-Ethnic 52 56
Other 18 21
Testing Location
In-Person 839 813
.745 0.044Flexible 18 18
Mobile 303 324
Not Selected 101 106
Group Subgroup
Comparison
Students
Imagine Math
Students p-value SMD
n 1,244 1,244
Average (SD) Fall
2020 NWEA MAP
Growth Math RIT
Score
193.62 (12.21) 194.12 (12.29) .303 0.041
Gender Female 641 644 .936 0.005Male 603 600
Race/Ethnicity
White
(Non-Hispanic)
663 638
.809 0.061
Asian 69 69
Black 257 257
Hispanic 172 185
Multi-Ethnic 65 70
Other 18 25
Testing Location
In-Person 871 802
.030 0.120Flexible 16 18
Mobile 257 299
Not Selected 100 125
Table C4. Grade 3 Baseline Equivalence
14Impact Evaluation of Imagine Math in a Charter School Network
Table C7. Grade 6 Baseline Equivalence
Group Subgroup
Comparison
Students
Imagine Math
Students p-value SMD
n 970 970
Average (SD) Fall
2020 NWEA MAP
Growth Math RIT
Score
217.91 (12.65) 218.12 (12.51) .713 0.017
Gender Female 508 502 .820 0.012Male 462 468
Race/Ethnicity
White
(Non-Hispanic)
440 434
.961 0.046
Asian 38 40
Black 192 193
Hispanic 256 250
Multi-Ethnic 43 52
Other 1 1
Testing Location
In-Person 600 589
.920 0.032Flexible 21 19
Mobile 263 270
Not Selected 86 92
Group Subgroup
Comparison
Students
Imagine Math
Students p-value SMD
n 1,166 1,166
Average (SD) Fall
2020 NWEA MAP
Growth Math RIT
Score
214.56 (12.81) 214.84 (12.60) .594 0.022
Gender Female 610 601 .740 0.015Male 556 565
Race/Ethnicity
White
(Non-Hispanic)
573 546
.400 0.094
Asian 51 63
Black 282 274
Hispanic 198 201
Multi-Ethnic 50 63
Other 12 19
Testing Location
In-Person 734 707
.694 0.050Flexible 24 28
Mobile 320 339
Not Selected 88 92
Table C6. Grade 5 Baseline Equivalence
15Impact Evaluation of Imagine Math in a Charter School Network
Table C9. Grade 8 Baseline Equivalence
Group Subgroup
Comparison
Students
Imagine Math
Students p-value SMD
n 656 656
Average (SD) Fall
2020 NWEA MAP
Growth Math RIT
Score
227.45 (15.96) 227.73 (16.19) .753 0.017
Gender Female 338 346 .699 0.024Male 318 310
Race/Ethnicity
White
(Non-Hispanic)
371 347
.309 0.121
Asian 14 26
Black 115 120
Hispanic 126 129
Multi-Ethnic 30 34
Other 0 0
Testing Location
In-Person 348 344
.959 0.031Flexible 11 10
Mobile 223 222
Not Selected 74 80
Group Subgroup
Comparison
Students
Imagine Math
Students p-value SMD
900 900
Average () Fall
2020 NWEA MAP
Growth Math RIT
Score
223.96 (14.51) 224.28 (14.79) .647 0.022
Gender Female 478 472 .813 0.013Male 422 428
Race/Ethnicity
White
(Non-Hispanic)
460 434
.158 0.133
Asian 17 27
Black 183 176
Hispanic 211 215
Multi-Ethnic 26 44
Other 3 4
Testing Location
In-Person 506 485
.542 0.069Flexible 10 16
Mobile 289 298
Not Selected 95 101
Table C8. Grade 7 Baseline Equivalence
16Impact Evaluation of Imagine Math in a Charter School Network
Appendix D
Table D1. Kindergarten Regression Results
Coefficient Estimate Standard Error p-value
Intercept 91.39 3.04 <.001
Imagine Math User Indicator 0.93 0.51 .066
Fall 2020 MAP Growth RIT Score 0.50 0.02 <.001
Male Indicator 0.99 0.51 .052
Race/Ethnicity
Indicator
Asian 5.07 1.19 <.001
Black -3.58 0.78 <.001
Hispanic -1.22 0.63 .051
Multi-Ethnic -0.38 1.10 .728
Other 1.52 7.72 .844
Testing Location Flexible 1.81 3.88 .641
Mobile 0.69 0.71 .329
Not Selected -0.13 0.83 .872
Coefficient Estimate Standard Error p-value
Intercept 88.57 3.44 .001
Imagine Math User Indicator 1.88 0.52 <.001
Fall 2020 MAP Growth RIT Score 0.55 0.02 <.001
Male Indicator 1.49 0.52 .005
Race/Ethnicity
Indicator
Asian 6.13 1.33 <.001
Black -2.95 0.81 <.001
Hispanic -1.98 0.63 .002
Multi-Ethnic -0.39 1.09 .720
Other -6.88 3.59 .055
Testing Location Flexible -6.79 3.59 .059
Mobile 0.12 0.65 .854
Not Selected -2.60 0.87 .003
Table D2. Grade 1 Regression Results
17Impact Evaluation of Imagine Math in a Charter School Network
Table D3. Grade 2 Regression Results
Coefficient Estimate Standard Error p-value
Intercept 96.34 3.16 <.001
Imagine Math User Indicator 2.59 0.44 <.001
Fall 2020 MAP Growth RIT Score 0.53 0.02 <.001
Male Indicator 1.27 0.45 .004
Race/Ethnicity
Indicator
Asian 5.26 1.04 <.001
Black -3.03 0.55 <.001
Hispanic -2.79 0.66 <.001
Multi-Ethnic 2.22 1.16 .056
Other -0.02 2.45 .993
Testing Location Flexible -2.62 3.25 .420
Mobile -2.29 0.53 <.001
Not Selected -1.28 0.72 .078
Coefficient Estimate Standard Error p-value
Intercept 82.04 2.94 <.001
Imagine Math User Indicator 1.77 0.36 <.001
Fall 2020 MAP Growth RIT Score 0.64 0.02 <.001
Male Indicator 0.36 0.36 .317
Race/Ethnicity
Indicator
Asian 1.38 0.81 .089
Black -3.43 0.46 <.001
Hispanic -0.78 0.53 .141
Multi-Ethnic -0.68 0.80 .397
Other -0.60 1.37 .661
Testing Location Flexible -4.17 1.54 .007
Mobile -2.46 0.45 <.001
Not Selected -2.48 0.63 <.001
Table D4. Grade 3 Regression Results
18Impact Evaluation of Imagine Math in a Charter School Network
Table D5. Grade 4 Regression Results
Coefficient Estimate Standard Error p-value
Intercept 44.49 3.19 <.001
Imagine Math User Indicator 1.84 0.35 <.001
Fall 2020 MAP Growth RIT Score 0.83 0.02 <.001
Male Indicator 0.79 0.35 .024
Race/Ethnicity
Indicator
Asian 1.61 0.78 .038
Black -2.40 0.45 <.001
Hispanic -0.53 0.46 .254
Multi-Ethnic -0.09 0.88 .914
Other -0.42 1.42 .766
Testing Location Flexible -0.81 1.47 .581
Mobile -2.24 0.42 <.001
Not Selected 0.49 0.64 .450
Coefficient Estimate Standard Error p-value
Intercept 17.54 3.06 <.001
Imagine Math User Indicator 1.36 0.35 <.001
Fall 2020 MAP Growth RIT Score 0.96 0.01 <.001
Male Indicator 1.12 0.35 .001
Race/Ethnicity
Indicator
Asian 2.99 0.84 <.001
Black -2.13 0.44 <.001
Hispanic -0.05 0.49 .922
Multi-Ethnic -0.10 0.83 .902
Other -1.80 1.53 .238
Testing Location Flexible -0.88 1.19 .459
Mobile -2.38 0.41 <.001
Not Selected -1.28 0.66 .055
Table D6. Grade 5 Regression Results
19Impact Evaluation of Imagine Math in a Charter School Network
Table D7. Grade 6 Regression Results
Coefficient Estimate Standard Error p-value
Intercept 19.33 3.60 <.001
Imagine Math User Indicator 2.04 0.39 <.001
Fall 2020 MAP Growth RIT Score 0.94 0.02 <.001
Male Indicator 0.48 0.40 .229
Race/Ethnicity
Indicator
Asian 2.13 1.03 .039
Black -1.51 0.53 .005
Hispanic 0.34 0.49 .488
Multi-Ethnic 0.04 0.94 .968
Other -5.58 6.14 .364
Testing Location Flexible 0.52 1.40 .709
Mobile -2.21 0.46 <.001
Not Selected 0.67 0.70 .339
Coefficient Estimate Standard Error p-value
Intercept 20.47 3.71 <.001
Imagine Math User Indicator 1.94 0.46 <.001
Fall 2020 MAP Growth RIT Score 0.93 0.02 <.001
Male Indicator 0.85 0.46 .068
Race/Ethnicity
Indicator
Asian -0.61 1.52 .688
Black -1.94 0.62 .002
Hispanic -0.27 0.58 .646
Multi-Ethnic -0.47 1.22 .701
Other -1.46 3.73 .696
Testing Location Flexible 1.24 1.95 .526
Mobile -0.69 0.52 .180
Not Selected 0.76 0.77 .322
Table D8. Grade 7 Regression Results
20Impact Evaluation of Imagine Math in a Charter School Network
Coefficient Estimate Standard Error p-value
Intercept 20.29 5.13 <.001
Imagine Math User Indicator 4.59 0.69 <.001
Fall 2020 MAP Growth RIT Score 0.95 0.02 <.001
Male Indicator 0.21 0.69 .756
Race/Ethnicity
Indicator
Asian 1.54 2.04 .450
Black -3.61 0.95 <.001
Hispanic -0.46 0.92 .618
Multi-Ethnic -2.01 1.62 .216
Other
Testing Location Flexible 1.42 2.76 .608
Mobile -0.88 0.77 .250
Not Selected -3.41 1.12 .002
Table D9. Grade 8 Regression Results
21Impact Evaluation of Imagine Math in a Charter School Network
Notes:
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
22Impact Evaluation of Imagine Math in a Charter School Network
Notes:
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
23Impact Evaluation of Imagine Math in a Charter School Network