of 47

Current View



Impact Evaluation of Imagine
Learning Illustrative Mathematics

in Fort Zumwalt School District

Michael A. Cook, PhD

Jane Eisinger, MS

Steven M. Ross, PhD

August 2023
ii

© Johns Hopkins University, 2023

Contents



EXECUTIVE SUMMARY ............................................................................................. iii

Impact Evaluation of Imagine Learning Illustrative Mathematics in Elementary and
Middle School Classes ............................................................................................... iii

Program Description ............................................................................................. iii

Research Design ................................................................................................... iii

Study Sample ....................................................................................................... iv

Program Impact on Mathematics Achievement
....................................................... iv
Teacher Perceptions
............................................................................................. iv
Impact Evaluation of Imagine Learning Illustrative Math in Elementary and Middle
School Classes .......................................................................................................... 1

Method ..................................................................................................................... 2

Research Design .................................................................................................... 2

Participants ........................................................................................................... 2

Quantitative sample. ........................................................................................... 2

Teacher sample. ................................................................................................. 3

Measures .............................................................................................................. 3

Galileo Comprehensive Assessment System Data.................................................. 4

Missouri Assessment Program. ............................................................................ 4

Demographics and rostering. .............................................................................. 4

Teacher questionnaire. ....................................................................................... 5

Analytical Approach ............................................................................................... 5

Achievement Results ................................................................................................. 5

Grade 6 descriptives. .......................................................................................... 6

Impact on Elementary Student Mathematics Achievement ........................................ 6

Impact analyses. ................................................................................................ 6

Subgroup analyses. ............................................................................................ 8

Impact on Middle School Mathematics Achievement................................................. 9

Impact analyses. ................................................................................................ 9

Subgroup analyses. .......................................................................................... 11

Teacher Questionnaire Results ................................................................................. 12

© Johns Hopkins University, 2023

Background ......................................................................................................... 12

Professional Development .................................................................................... 13

Curriculum Implementation .................................................................................. 15

Perceived Impact on Student Learning .................................................................. 19

Overall Perceptions .............................................................................................. 22

Discussion .............................................................................................................. 23

Conclusions ......................................................................................................... 25

Appendix A: Teacher Questionnaire.......................................................................... 26

Appendix B: Baseline Equivalence Tables .................................................................. 32

Appendix C: Descriptive Achievement Tables ............................................................ 33

Appendix D: MAP Subscale Results ........................................................................... 35

Appendix E: Subgroup Regression Tables Elementary ............................................. 36

Appendix F: Subgroup Regression Tables Middle .................................................... 39



EXECUTIVE SUMMARY iii
© Johns Hopkins University, 2023

EXECUTIVE SUMMARY

Impact Evaluation of Imagine Learning Illustrative Mathematics in
Elementary and Middle School Classes

Program Description


Imagine Learning’s Illustrative Mathematics (IL Illustrative Math) is a problem-
based core curriculum for K12 students. It is designed to be used in face-to-face
instruction in student-led whole group instruction. Per the developer, Illustrative
Mathematics (IM) is a K12® core curriculum designed to give all students equity and
access to grade-level mathematicsensuring each student is an active participant in
their learning. IL Illustrative Math is a problem-based curriculum that is designed to
provide conceptual understanding, procedural fluency, strategic competence, adaptive
reasoning, and a productive disposition. Students learn by doing, working their way
through problems in both mathematical and real-world contexts and constructing
arguments using precise language.


Research Design

Imagine Learning contracted with the Center for Research and Reform in
Education (CRRE) at Johns Hopkins University to conduct a quasi-experimental design
(QED) in the Fort Zumwalt School District (FZSD) in O’Fallon, MO. FZSD is the largest
school district in St. Charles County, serving approximately 18,500 students in 16
elementary, four middle, and four high schools. Grade-level teachers in elementary and
middle schools were given the option of implementing IL Illustrative Math in the 2022
23 school year. At the elementary school level (Grades K5), participation counts across
the 16 schools were 149 classrooms in the intervention group and 147 in the
comparison (business-as-usual) group. Nearly all grades and schools across the 16
elementary schools contained a mix of classrooms that used or did not use IL
Illustrative Math, providing for an ideal comparison group and eliminating potential
confounding variables. At the middle school level, considerably larger numbers of
classrooms used IL Illustrative Math, especially in Grade 6, where nearly all classrooms
used the program. Thus, elementary classrooms were analyzed separately from middle
school classrooms, and Grade 6 students were only analyzed descriptively, as nearly
95% of students participated in using IL Illustrative Math.


The evaluation also examined teachers’ perceptions of IL Illustrative Math
through an online teacher questionnaire. Teachers were asked about topics including
instructional practices, IL Illustrative Math curriculum implementation, professional
development, and student impact. Likert-scale items were used to collect data relating
to teachers’ perceptions of IL Illustrative Math. Additionally, four open-ended queries
provided teachers with the opportunity to describe curriculum use including any
EXECUTIVE SUMMARY iv

© Johns Hopkins University, 2023

deviations they made and advice they would offer to teachers new to IL Illustrative
Math.


Study Sample


The present study sample included 6,476 Grades 38 students across 16
elementary schools and four middle schools. The vast majority of students (around
85% in both conditions) were White. Teacher questionnaire data were collected from
227 teachers and administrators.


Program Impact on Mathematics Achievement


A significant positive impact of IL Illustrative Math on student mathematics
achievement was observed for middle school students in Grades 7 and 8. Treatment
students who received the IL Illustrative Math program made nearly 16-point larger
gains on the
Galileo Comprehensive Assessment System (GCAS) mathematics scores
from beginning-of-year (BOY) to end-of-year (EOY) of the 202223 school year than did
comparison students. In addition, special education students in middle school
significantly outgained their special education comparison counterparts from BOY to
EOY. Significant positive IL Illustrative Math impacts were also observed for Grades 4
and 5 students on the Missouri Assessment Program (MAP) mathematics assessment,
with treatment (IL Illustrative Math) students outscoring comparison students by nearly
11 points. As with middle school GCAS scores, special education students in elementary
school significantly outgained their special education comparison counterparts on the
MAP mathematics assessment by 13 points from spring 2022 to spring 2023.


Teacher Perceptions


Teacher perceptions of IL Illustrative Math were generally very positive,
especially regarding perceptions of program impacts on multiple aspects of student
learning in math, with more than 90% of program teachers agreeing that the
curriculum:


appropriately challenged students during math instruction
led to student engagement in high-level discussions during math instruction
improved students’ ability to work in groups
improved students’ ability to problem solve mathematically
Program teachers also expressed very positive overall perceptions of IL
Illustrative Math, with nearly 90% of teachers agreeing that they would recommend
the program to other teachers. Teacher perceptions of professional development were
generally positive, although agreement was slightly lower for the digital tools and
resources versus the other curriculum components. Overall, teachers generally
EXECUTIVE SUMMARY v

© Johns Hopkins University, 2023

perceived the curriculum very positively, but they were sometimes challenged to
implement all features of the program during their allocated math blocks.



Imagine Learning Illustrative Math Fort Zumwalt 1
© Johns Hopkins University, 2023

Impact Evaluation of Imagine Learning Illustrative Math in
Elementary and Middle School Classes

Imagine Learning contracted with the Center for Research and Reform in
Education (CRRE) at Johns Hopkins University to conduct a quasi-experimental design
(QED) in the Fort Zumwalt School District (FZSD) in O’Fallon, MO. FZSD is the largest
school district in St. Charles County, serving approximately 18,500 students in 16
elementary, four middle and four high schools. Grade-level teachers in elementary and
middle schools were given the option of implementing IL Illustrative Math in the 2022
23 school year. At the elementary school level (Grades K5), participation counts across
the 16 schools were 149 classrooms in the intervention group and 147 in the
comparison (business-as-usual) group. In middle school, nearly all Grade 6 classrooms
implemented IL Illustrative Math, while three of four schools used IL Illustrative Math in
Grade 7. Grade 8 contained a mix of treatment and comparison students across all four
FZSD middle schools. In only three of the 16 elementary schools did all grades and
classes use one treatment, specifically here, the intervention. The remainder had mixed
treatment participation across grades, thereby eliminating contextual confounding risks.
It is important to note that classrooms that implemented IL Illustrative Math were doing
so for the first time.


IL Illustrative Math is a problem-based core curriculum for K12 students. It is
designed to be used in face-to-face instruction in student-led whole group instruction.
Per the developer, “Illustrative Mathematics is a K12® core curriculum designed to
give all students equity and access to grade-level mathematicsensuring each student
is an active participant in their learning. IL Illustrative Math is designed to provide
conceptual understanding, procedural fluency, strategic competence, adaptive
reasoning, and a productive disposition. Students learn by doing, working their way
through problems in both mathematical and real-world contexts and constructing
arguments using precise language.”


The evaluation’s design addressed the following research questions:


1) How does participation in IL Illustrative Math impact student achievement in
mathematics?

a) Does level of program usage relate to student achievement effects?

b) To what degree do effects vary across:

i) Schools

ii) Grade levels

iii) Student subgroups (ethnicity, ELL, SPED, FARMS)

2) What are teachers’ perceptions of the IL Illustrative Math program with regard
to:

a) Benefits for students?

b) Student engagement?

c) Implementation requirements?
Imagine Learning Illustrative Math Fort Zumwalt 2
© Johns Hopkins University, 2023

d) Strengths and weaknesses?

e) Recommendations for implementation improvement?


Method

Research Design

The current study used a quasi-experimental design (QED) in 20 schools in the
Fort Zumwalt School District (FZSD) in O’Fallon, MO. FZSD is the largest school district
in St. Charles County, serving approximately 18,500 students in 16 elementary, four
middle and four high schools. Grade-level teachers in elementary and middle schools
were given the option of implementing IL Illustrative Math in the 202223 school year.
Nearly 6,000 students comprised the overall analytic sample, which is described in more
detail below.


Qualitative data were collected through an online teacher questionnaire that was
administered to all intervention teachers. Likert-scale questionnaire items were analyzed
descriptively, while open-ended responses were analyzed using qualitative analytic
techniques (Miles, Huberman, & Saldana, 2002).


Participants


Quantitative sample. Grades 38 students were included in quantitative
achievement analyses if they had non-missing pretest (BOY) and posttest (EOY for
Grades 35, MOY for Grades 68) Galileo scores (see description below), as well as
demographic data. As Grades 35 students and Grades 68 students were considered
as distinct analytic samples, we present demographic breakdowns of treatment and
comparison samples by analytic sample in Table 1.


Table 1


Student Demographics (Percentages), by Grade Range


Subgroup
Treatment Comparison
Grades 35

Female
47.28 48.67
White
84.90 85.30
Black
7.15 7.40
Hispanic
5.01 7.03
Asian
4.97 4.25
Other Race
2.98 3.05
SPED
17.35 14.64
ELL
5.54 5.77
FARMS
19.07 20.41
Imagine Learning Illustrative Math Fort Zumwalt 3
© Johns Hopkins University, 2023

N
1510 1837
Grades 68

Female
49.00 46.10
White
83.76 84.00
Black
7.91 12.57
Hispanic
6.80 7.05
Asian
4.76 0.57
Other Race
3.57 2.86
SPED
9.98 34.86*
ELL
6.57 2.77
FARMS
18.82 25.71
N
2604 525
Note. * p < .05.


Elementary (Grades 35) students were very similar on nearly all demographic
variables, with only small discrepancies observed in percentages of Hispanic and SPED
students. Middle school (Grades 68) students were considerably different in terms of
special education, with a much larger percentage (35%) of comparison students
identified as needing special education services, in relation to only 10% of treatment
students. The middle school comparison group also contained larger but not
significantly different proportions of free and reduced meals students (FARMS). It is
important to note that, while the elementary sample size is relatively evenly split
between treatment and comparison, a vast majority of the middle school sample (83%)
received IL Illustrative Math. This was expected, as all Grade 6 classrooms and nearly
all Grade 7 classrooms were initially expected to use IL Illustrative Math at the
beginning of the school year.


Teacher sample. A total of 250 treatment teachers across Grades K8 in 16
elementary schools and four middle schools were invited to complete the questionnaire
and were offered an incentive in the form of a $15 gift card for their participation. A
total of 227 participants completed the questionnaire, yielding an extremely high 90.8%
response rate. Most participants were classroom teachers, with smaller numbers of
special education teachers, instructional coaches, and school principals providing
feedback.


Measures


Data sources for the current study include Galileo Comprehensive Assessment
System mathematics scores from the 202223 school year, along with Missouri
Assessment Program mathematics scores from the spring of both 2022 and 2023, and
demographic data provided by FZSD. Teacher data included Likert-scale questionnaire
items relating to perceptions of the IL Illustrative Math program, as well as free-
response and binary-choice items.

Imagine Learning Illustrative Math Fort Zumwalt 4
© Johns Hopkins University, 2023

Galileo Comprehensive Assessment System Data. Imagine Learning
provided CRRE with GCAS mathematics score data from the BOY, MOY, and EOY of the
202223 school year for all FZSD Grades 38 students. The GCAS was created by
Assessment Technology Incorporated (ATI)1 and is designed for Grades K12 students.
The GCAS is a computer-administered progress-monitoring assessment in English
Language Arts (ELA), mathematics, and science. The GCAS can be administered up to
three times a year, although FZSD administered the GCAS only at BOY and EOY for
elementary students, while administering the GCAS at all three timepoints to middle
school students. It is important to note that the EOY GCAS assessment administered by
FZSD is more cumulative in nature than the BOY and MOY tests for middle school
grades, differing from the original design of the assessments. Thus, EOY scores may
have slightly different interpretations than BOY and MOY scores for middle grades.
Further, Imagine Learning and FZSD reported variation in how the EOY assessment was
administered, especially in terms of content coverage. After discussion with Imagine
Learning and FZSD, the decision was made to use MOY GCAS scores as the outcome
variable for middle school analyses, while using EOY GCAS scores as the outcome
variable for elementary school analyses.


Missouri Assessment Program. FZSD provided CRRE with spring 2022 and
2023 Missouri Assessment Program (MAP) mathematics scores. According to the
Missouri Department of Elementary and Secondary Education2, the MAP is designed to
measure how well students acquire the skills and knowledge described in the Missouri
Learning Standards (MLS). The MAP is administered in the spring of each school year to
students in Grades 38 in ELA and mathematics, with the science MAP assessment
administered to Grades 5 and 8 students. The MAP mathematics assessment is
administered in a secure online format over the course of three sessions. MAP scores
are vertically scaled, meaning that scores from students in different grades can be
directly compared. In the 202223 school year, MAP mathematics scores ranged from
185660. Spring 2023 MAP mathematics scores were used as outcome variables in
impact analyses, while spring 2022 MAP mathematics scores were used as a prior
mathematics achievement variable.


Demographics and rostering. FZSD provided CRRE with demographic and
rostering data from the 202223 school year. Demographic variables included student
grade level, ethnicity, gender, Individualized Education Program (IEP) status and codes,
language spoken at home (other than English was classified as English language learner
(ELL)), race, and FARMS. Rostering data were supplied at BOY, MOY, and EOY for
elementary and middle school students. These lists were checked for potential condition
movement among students. As minimal condition mobility was observed, rostering data
from the same time point as outcome variable collection (i.e., EOY for elementary

1 Imagine Learning recently acquired ATI. The GCAS system is now a part of Imagine Learning. Imagine
Learning has not completed any alignment efforts at the time of this report.
2 https://dese.mo.gov/special-education/effective-practices/student-assessments/missouri-assessment-
program.
Imagine Learning Illustrative Math Fort Zumwalt 5
© Johns Hopkins University, 2023

students and MOY for middle school students) were used for condition assignment
purposes.


Teacher questionnaire. The teacher questionnaire was administered to
teachers of IL Illustrative Math students in the 202223 school year. The questionnaire
included curriculum-specific questions relating to classroom practices, student
motivation and achievement, program implementation and usage, professional
development and training, and overall program perceptions. The questionnaire
contained Likert-scale and yes/no questions, along with four open-ended items. Likert-
scale questionnaire responses were analyzed using descriptive statistics (e.g.,
percentages and counts), while open-ended questionnaire responses were analyzed
qualitatively. A copy of the teacher questionnaire can be found in Appendix A.


Analytical Approach


Hierarchical Linear Modeling (HLM) was used to examine the impact of IL
Illustrative Math on student math achievement gains. As discussed earlier, elementary
and middle school grades were considered separately due to differences in program
implementation, as well as available outcome data. In addition, the middle school
analytic sample for these analyses consisted of only Grades 78, as Grade 6 was
comprised of 95% treatment students. Baseline equivalence was met for the
elementary sample, with a standardized mean difference of 0.04 SDs. Baseline
equivalence was also met with the middle school (Grades 78) sample, with a
standardized mean difference of 0.12 SDs. No statistical matching techniques were
used in these analyses, given the fulfillment of the baseline equivalence requirement, as
well as sampling limitations, especially in the middle school grades. Full baseline
equivalence tables can be found in Appendix B. As described earlier, for the GCAS
mathematics assessment, EOY scores were considered the outcome variable of interest
for elementary students, while MOY scores were considered the outcome variable of
interest for middle school students. Thus, two sets of analyses were conducted for the
GCAS: one each for elementary and middle school samples. Similarly, spring 2023 MAP
mathematics scores were considered the outcome variables for MAP analyses, with
spring 2022 MAP mathematics scores used as the prior achievement control. The
analytic approach for the main impact analyses was the same across elementary and
middle grades, and for both GCAS and MAP analyses.


Achievement Results

We begin by descriptively examining Grade 6 scores, which were not included in
the impact analyses due to nearly all Grade 6 students receiving the IL Illustrative Math
program. Analyses examining the impacts of IL Illustrative Math on GCAS and MAP
mathematics scores in other grades follow. Subgroup analyses were also conducted
examining IL Illustrative Math’s impacts on student subgroups of interest.

Imagine Learning Illustrative Math Fort Zumwalt 6
© Johns Hopkins University, 2023

Grade 6 descriptives. A total of 1,068 of the 1,115, or 95.8% of Grade 6
students with non-missing BOY and MOY GCAS mathematics scores participated in IL
Illustrative Math in the 2022-23 school year. For this reason, we limited our analyses of
Grade 6 students to descriptive trend analyses. Table 2 shows average BOY and MOY
GCAS and MAP mathematics scores for IL Illustrative Math and comparison Grade 6
students.


Table 2


Grade 6 GCAS and MAP Mathematics Scores


BOY MOY N Growth
GCAS

IL Illustrative Math
1069.67 (75.44) 1153.75 (80.71) 1,068 84.08
Comparison
972.36 (66.54) 1007.64 (81.47) 47 35.28
MAP

IL Illustrative Math
413.28 (30.44) 425.46 (30.33) 1,037 12.18
Comparison
334.98 (33.63) 361.98 (35.33) 47 27.00
Note. SD in parentheses.


On the GCAS, IL Illustrative Math students gained approximately 84 points from
BOY to MOY, in relation to comparison students, who averaged about 35-point gains.
This represents a considerable advantage for the IL Illustrative Math condition, as gains
in this condition were nearly 50 points larger than those for comparison students. The
pattern was reversed for the MAP assessment, though, with comparison students
outgaining IL Illustrative Math students by nearly 15 points. However, with such a small
number of comparison students present in Grade 6, and with comparison students
having BOY scores more than 1 standard deviation lower than that for treatment
students on both assessments, these trends should be interpreted with considerable
caution. Full descriptive tables across all grades can be found in Appendix C.


Impact on Elementary Student Mathematics Achievement


In this section, we discuss the results of the main impact analyses examining the
effect of IL Illustrative Math on elementary mathematics achievement, as measured by
the GCAS and MAP assessments. We will also examine the results of subgroup analyses
on both outcome measures.


Impact analyses. Table 3 shows the results of the analyses examining the
impact of IL Illustrative Math on EOY GCAS mathematics scores. Hierarchical linear
modeling (HLM) with students nested within classrooms was used for both main impact
analyses.


Imagine Learning Illustrative Math Fort Zumwalt 7
© Johns Hopkins University, 2023

Table 3


Overall Impact of IL Illustrative Math on Spring 2023 GCAS Mathematics Scores, Grades
35


Variable
Estimate
Standard
Error
p value
Effect
Size

IL Illustrative Math
-3.718 5.182 .473 -0.03
Constant
935.414*** 3.424 <.001
Variance of constant
898.601
Residual
5189.791
Student N
3106
Class N
194
Note. *** p < .001.


IL Illustrative Math students averaged slightly less than 4-point smaller gains on
the GCAS assessment from BOY to EOY than did comparison students. The large p
value and small effect size indicate that this difference in mathematics achievement
gains is statistically and practically small, meaning that patterns of achievement gains
were similar for students in both conditions.


Tables 4 and 5 show the results of analyses examining the impact of IL
Illustrative Math on spring 2023 MAP mathematics scores. Due to differences in
available prior achievement measures, separate analyses were conducted for Grade 3
students and Grades 4
5 students. Impact analyses similar to those used in the prior
analysis were conducted here, with spring 2022 MAP score used as the prior
achievement variable for Grades 4
5, while the BOY GCAS score was used as the prior
achievement variable for the Grade 3 analysis. The analyses below examined overall
composite MAP mathematics scores; MAP subscale score analyses are presented in
Appendix D.


Table 4


Overall Impact of IL Illustrative Math on Spring 2023 MAP Mathematics Scores, Grades
45


Variable
Estimate
Standard
Error
p value
Effect
Size

IL Illustrative Math
10.907** 3.869 .005 0.28
Constant
396.78*** 1.960 <.001
Variance of constant
77.045
Residual
645.525
Student N
1,994
Class N
128
Imagine Learning Illustrative Math Fort Zumwalt 8
© Johns Hopkins University, 2023

Note. ** p < .01; *** p < .001.


Table 5


Overall Impact of IL Illustrative Math on Spring 2023 MAP Mathematics Scores, Grade 3


Variable
Estimate
Standard
Error
p value
Effect
Size

IL Illustrative Math
24.358 14.601 .095 0.57
Constant
341.407*** 7.337 <.001
Variance of constant
62.086
Residual
803.712
Student N
1,022
Class N
69
Note. *** p < .001.


Significant positive impacts of IL Illustrative Math were evidenced for Grades 4
5
students, with treatment students in these grades outgaining comparison students by
nearly 11 points. A positive program impact was also evidenced in Grade 3, with
treatment students outgaining comparison students by 24 points, but this impact did
not quite reach statistical significance (p < .10). Evidence of practical program impacts
on MAP achievement gains were evidenced across all elementary students, with effect
sizes of 0.28 SDs for Grades 4
5 students and 0.57 SDs for Grade 3 students. In
addition, MAP subscale score analyses found significant positive program impacts of IL
Illustrative Math on the Geometry (p < .01), Number Sense (Fractions), and
Relationships/Algebraic Thinking (p < .05 on each) subscales, with advantages of 13
16 points for treatment students on each of these subscales. In all, impacts of IL
Illustrative Math in elementary schools were much more positive on MAP mathematics
scores than on GCAS mathematics scores.


Subgroup analyses. Subgroup analyses were conducted to examine the impact
of IL Illustrative Math on elementary school subgroups of interest, including grade
levels, special education students, ELLs, FARMS students, and ethnicity. These analyses
consist of the model from the main impact analysis, along with product terms
estimating the interaction between the treatment variable and subgroup dummy
variable indicators. The additive effect of the treatment and treatment-by-subgroup
terms were tested to allow for the examination of the unique impact of IL Illustrative
Math within each subgroup of interest, as shown in Tables 6 and 7. Full tables of
subgroup analysis regression tables can be found in Appendix E. Note that MAP
subgroup analyses are only reported for Grades 4
5 students3.


3 Grade 3 was excluded from reporting due to small subgroup sample sizes. No significant subgroup
impacts in Grade 3 were found.
Imagine Learning Illustrative Math Fort Zumwalt 9
© Johns Hopkins University, 2023

Table 6


IL Illustrative Math Impact on Elementary GCAS Achievement, by Subgroup


Subgroup
Estimate p value N
Grade 3
5.163 .547 1023
Grade 4
-11.313 .214 1037
Grade 5
-6.133 .500 1046
Black
-0.357 .975 205
Hispanic
12.395 .303 194
SPED
9.067 .254 502
ELL
-8.368 .496 174
FARMS
4.116 .588 599

Table 7


IL Illustrative Math Impact on Elementary MAP Achievement, by Subgroup (Grades 4
5)

Subgroup
Estimate p value N
Grade 4
3.129 .493 991
Grade 5
16.201** .001 1,003
Black
5.320 .392 114
Hispanic
6.555 .315 121
SPED
13.433** .004 307
ELL
7.714 .231 112
FARMS
9.852* .036 358
Note. * p < .05; ** p < .01.


No significant subgroup impacts were evidenced when examining GCAS
mathematics score gains.
However, significant IL Illustrative Math program impacts on
MAP achievement gains were evidenced for Grade 5 students, special education
students, and FARMS students. The magnitudes of these advantages ranged from 10

16 points.


Impact on Middle School Mathematics Achievement

In this section, we discuss the results of the main impact analyses examining the
effect of IL Illustrative Math on middle school mathematics achievement, as measured
by the GCAS and MAP assessments. We will also examine the results of subgroup
analyses on both outcome measures.


Impact analyses. Tables 8 and 9 show the results of the analyses examining
the impact of IL Illustrative Math on MOY GCAS and spring 2023 MAP mathematics
scores.
Imagine Learning Illustrative Math Fort Zumwalt 10
© Johns Hopkins University, 2023


Table 8


Overall Impact of IL Illustrative Math on Winter 2023 GCAS Mathematics Scores, Grades
78


Variable
Estimate
Standard
Error
p value
Effect
Size

IL Illustrative Math
15.972* 7.301 .029 0.16
Constant
1225.460*** 6.686 <.001
Variance of constant
465.553
Residual
4596.061
Student N
1875
Class N
36 Note. * p < .05; *** p < .001.

The impact of IL Illustrative Math on middle school GCAS mathematics
achievement gains was statistically significant (p = .029), with IL Illustrative Math
students averaging nearly 16-point larger gains from BOY to MOY than did comparison
students. The effect size of this impact was 0.16 SDs, indicating a small-to-medium
practical effect of IL Illustrative Math on mathematics achievement.


Table 9


Overall Impact of IL Illustrative Math on Spring 2023 MAP Mathematics Scores, Grades
78


Variable
Estimate
Standard
Error
p value
Effect
Size

IL Illustrative Math
-1.068 2.947 .717 -0.02
Constant
450.238*** 2.532 <.001
Variance of constant
37.696
Residual
667.928
Student N
1,841
Class N
36
Note. *** p < .001.


No significant impact of IL Illustrative Math on middle school MAP mathematics
achievement gains was evidenced. Comparison students slightly outgained treatment
students by approximately 1 point, with an effect size close to zero (.02 SDs). MAP
achievement gains therefore were comparable for IL Illustrative Math and comparison
students. While no significant program impacts were evidenced on overall MAP
mathematics scores, a significant positive impact was found on the Geometry subscale,
with IL Illustrative Math students outscoring comparison students by nearly 7 points (p
Imagine Learning Illustrative Math Fort Zumwalt 11
© Johns Hopkins University, 2023

< .01). As with the elementary analyses, full results of analyses examining program
impacts on MAP mathematics subscale scores can be found in Appendix D.


Subgroup analyses. The same subgroups examined in the elementary
analyses were included in these analyses. Tables 10 and 11 show the impacts of IL
Illustrative Math across all student subgroups for the GCAS and MAP mathematics
assessments, respectively. Full tables of subgroup regression analyses can be found in
Appendix F.


Table 10


IL Illustrative Math Impact on Middle School GCAS Achievement, by Subgroup


Subgroup
Estimate p value N
Grade 7
17.338 .100 1086
Grade 8
14.566 .130 789
Black
2.965 .822 153
Hispanic
14.056 .387 130
SPED
26.258** .005 266
ELL
23.834 .296 106
FARMS
8.981 .371 358
Note. ** p < .01.


Table 11


IL Illustrative Math Impact on Middle School MAP Achievement, by Subgroup


Subgroup
Estimate p value N
Grade 7
-0.430 .917 1,066
Grade 8
-1.566 .667 775
Black
-4.616 .384 138
Hispanic
-3.809 .546 132
SPED
-1.723 .643 248
ELL
4.231 .634 102
FARMS
-3.967 .325 336

One statistically significant subgroup impact on the GCAS mathematics
assessment was observed for special education students. Special education students
who used IL Illustrative Math outgained special education comparison students by more
than 26 points (p = .005). IL Illustrative Math impacts were directionally positive across
all other subgroups on the GCAS mathematics assessment, although impacts did not
reach statistical significance. No significant program subgroup analyses were evidenced
on the MAP mathematics achievement.

Imagine Learning Illustrative Math Fort Zumwalt 12
© Johns Hopkins University, 2023

Teacher Questionnaire Results

Major takeaways from teacher questionnaire responses are presented in the
section below. We begin with findings pertaining to teacher backgrounds and IL
Illustrative Math curriculum implementation. These sections are followed by results on
perceived impact on student learning, professional development, and overall
perceptions of the curriculum.


Background


Table 12 shows the number of questionnaire participants by school. Respondents
(n = 227) represented 16 different elementary schools and four middle schools within
the district.


Table 12


Respondent Numbers by School


School Name

Number

of Respondents

Percentage of
Respondents

Dubray Middle School
5 2.20%
North Middle School
8 3.52%
South Middle School
9 3.96%
West Middle School
13 5.73%
Dardenne Elementary School
13 5.73%
Emge Elementary School
5 2.20%
Flint Hill Elementary School
11 4.85%
Forest Park Elementary School
15 6.61%
Hawthorn Elementary School
21 9.25%
JL Mudd Elementary
8 3.52%
Lewis & Clark Elementary School
8 3.52%
Mid Rivers Elementary School
5 2.20%
Mount Hope Elementary School
5 2.20%
Ostmann Elementary School
4 1.76%
Pheasant Point Elementary School
2 0.88%
Progress South Elementary School
34 14.98%
Rock Creek Elementary School
6 2.64%
St. Peters Elementary School
11 4.85%
Twin Chimneys Elementary School
16 7.05%
Westhoff Elementary School
28 12.33%
Total
227 100%
Imagine Learning Illustrative Math Fort Zumwalt 13
© Johns Hopkins University, 2023

Table 13 shows all grade(s) taught by participants. The majority of respondents
(87.3%) were from elementary schools.


Table 13


Respondent Numbers by Grade Level


Grade Level

Number of
Respondents/Grade
Percentage of Respondents
Kindergarten
31 11.61%
First Grade
37 13.86%
Second Grade
32 11.99%
Third Grade
49 18.35%
Fourth Grade
38 14.23%
Fifth Grade
46 17.23%
Sixth Grade
13 4.87%
Seventh Grade
14 5.24%
Eighth Grade
7 2.62%
Total
267 100%

The majority of participants (85.9%) identified primarily as classroom teachers (n
= 175) or special education (SPED) teachers (n = 20). Other participants included 17
coaches (7.5%) and two interventionists (0.8%). Additionally, seven school principals
and six self-described “administrators” also participated in the questionnaire. The
administrators and principals provided limited responses and their data was excluded
from the remainder of this analysis since their role in curriculum implementation is
unknown.


Professional Development


Teachers received professional development related to the
IL Illustrative Math
curriculum, as well as ongoing support in their implementation, at both the district and
school levels. Related questionnaire items aimed to evaluate the helpfulness of this
training and support. In particular, teachers were asked to indicate their level of
agreement with the following statements (see Figure 1). In this set of items (and all
remaining Likert-scale items), percent agreement is defined as the percentage of
teachers that somewhat agree or strongly agree with an item, while percent
disagreement is defined as the percentage of teachers who somewhat disagree or
strongly disagree with an item.


Imagine Learning Illustrative Math Fort Zumwalt 14
© Johns Hopkins University, 2023

Figure 1


Teacher Perceptions of
IL Illustrative Math Professional Development

Note. + < 5%.

Teachers generally agreed that they had received sufficient training to implement
both the curriculum’s digital tools and resources, as well as the curricular components
such as Warm-Ups and Cool Downs. However, slightly fewer teachers agreed that they
had received sufficient training for the digital piece than for the curricular components.


Teachers were asked to indicate their level of agreement with statements
regarding the adequacy of ongoing support that they received from their district and
from their school’s administration. Figure 2 depicts responses to these two items.


Figure 2


Teacher Ratings of District & School Administration Support


Note: + <5%.
+
+
+
14.8%
42.9%
44.8%
49.3%
35.5%
I received sufficient professional development to
successfully implement the various curriculum
components (e.g., Warm-Up, Activities, Cool-downs, etc.)
I received sufficient professional development to
successfully implement and utilize the digital tools and
resources available in the curriculum
Professional Development
Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree

5.9%
+
+
6.9%
32.0%
35.0%
57.1%
54.2%
I received adequate ongoing support from school
administration
I received adequate ongoing support from district
administration
Adequacy of Ongoing Teacher Support
Strongly disagree Somewhat disagree Somewhat agree Strongly agree
Imagine Learning Illustrative Math Fort Zumwalt 15
© Johns Hopkins University, 2023

There was strong agreement among teachers that they received adequate
ongoing support, with nearly 90% of teachers reporting that they received ongoing
support from both district and school sources, respectively.


Curriculum Implementation

Regarding the implementation of
IL Illustrative Math, roughly one-third of
teachers (38.6%) reported that they had used the curriculum prior to the 202223
school year, whereas roughly two-thirds of participants (61.4%) indicated that they had
not. Teachers were then asked to specify their usage frequency for specific curriculum
features when teaching math, either on paper or on the digital platform. Figure 3
displays these results.


Figure 3

Frequency of Usage for
IL Illustrative Math Program Features
Note. + < 5%.

Of the four features, teachers reported the highest frequency usage for the
Lesson Warm-Up and Lesson Cool Down features (91.1% and 73.9%, respectively,
used daily). Lesson Synthesis was also utilized by a majority of teachers (60.5%) each
day whereas Centers (a feature only in Grades K5) saw the lowest daily usage
(35.1%).


As part of the questionnaire, teachers were asked four open-ended questions in
order to provide more extensive feedback, in their own words, on several topics
regarding program implementation and satisfaction. The first of these queries was

+
+
+
15.5%
6.5%
9.9%
46.4%
31.0%
16.3%
5.5%
35.1%
60.5%
73.9%
91.1%
Centers (K-5 only)
Lesson Synthesis
Lesson Cool Down
Lesson Warm-Up
Usage of Program Features Per Week
0 1-2 days 3-4 days 5 days
Imagine Learning Illustrative Math Fort Zumwalt 16
© Johns Hopkins University, 2023

related to the use of program features, asking teachers if they had deviated from the
typical “flow” in the presentation of lessons and if so, how? Roughly one-third (n = 45,
29.6%) of the 152 teachers who responded indicated that they had not deviated from
the typical flow. Teacher comments included: “I did not deviate from this structure. It
effectively supported student learning,” and, “We did this flow and found that
sometimes cool downs made student[s] not want to start a lesson and cool downs
didn't always go with the lessons.” This second comment indicates that some of the
teachers who followed the typical flow did not find that it always worked optimally for
them. The remaining 107 participants (70.4%) who indicated that they had deviated
from the typical flow described the deviations they had made, and 70 participants
(46.1%) provided reasons for why they had done so. The majority of deviations took
one of the following two forms:


The teacher did less of/shortened the typical flow (n = 50, 32.9%)

The teacher re-ordered the typical flow (n = 46, 30.3%)


Teachers who described having done less than the prescribed lesson material in the
lessons, or having shortened the lesson, typically attributed the deviation to a lack of
time (n = 43, 86.0%). Teachers’ remarks on this included:


Centers was the only issue I have had this year. My struggle has been
trying to fit it all in and given this was our pilot year I often felt
overwhelmed doing all of the other things. However, next year I do not
foresee it being an issue.


Timing is the biggest challenge, especially in kindergarten. Some activities
took longer than what the book said. The reality of teaching kindergarten
is things take longer and we need to account for transitioning, behaviors,
directions, getting materials passed out, etc.


My math instructional time does not allow for the full lesson to be taught
in one chunk so we do our lesson for 10 minutes in the morning, followed
by centers, and then a few hours later we do the first two activities, break
for 20 min., recess and then do the synthesis and cool down.


The regular incorporation of Centers (a component of the Grades K5
curriculum) into the daily schedule was specifically noted as being particularly
challenging by some teachers (n =15, 9.9%). However, confronting a shortage of time
was not always perceived to be a negative, as shown in the following teacher’s
comment, “The deviation was not intentional as I had a difficult time moving on when
students were engaged in rich mathematical conversation which lead to me not getting
to centers every day.” Small numbers of teachers indicated that they had shortened
lessons or done “less” because of a variety of reasons including the need for more
student practice, because the lessons were “too hard”/students grew frustrated, or
Imagine Learning Illustrative Math Fort Zumwalt 17
© Johns Hopkins University, 2023

because the teachers felt the omitted material wasn’t needed/was redundant. Over two-
thirds (69.57%) of the teachers who reported having re-ordered the lesson flow said
this was because they moved Centers to being the first activity of the day. A small
number of teachers (n = 4, 8.7%) indicated this had been a school-wide or district-level
decision, as described in this teacher’s comment:


We decided with input from district math coordinators, to move center
time to be before starting lesson warm up. It worked as a "warm up"
before the warm up. It got students into partnerships and into math
thinking before the lesson begins.


Other teachers indicated that they preferred to do Centers first out of preference as
seen in these examples, “I follow the lesson flow as it is supposed to be done. Some
days I would start with centers as this worked better for some of the lessons,” and
“Sometimes we will do Centers at the beginning to get students into the math mindset.”
The most common additional reason given for the reordering of lessons was lack of
time (n = 9, 5.11%), with smaller numbers of teachers saying that it was a result of
their having combined activities or because they, as implementers, still needed to
improve their use of the curriculum.


A small number of teachers (n = 14, 9.2%) who provided answers other than
those outlined above stated they had deviated from the typical lesson flow for a variety
of reasons. Most commonly, teachers (n = 8, 5.2%) repeated units or added additional
practice opportunities for their students. Observations from these teachers included, “I
did more visuals such as filling out graphic organizers to help my visual learners,” and
“There were various units where the students needed more practice/repetition to access
skills from the previous year.” One special education teacher also commented, “Often
the activities were too hard. I teach in the Gen Ed but also in the SPED setting and the
activities were too hard for many of the students to do. They need more repetition and
explanation.” Overall, teachers appear to have followed the planned lesson flow for the
program with small numbers indicating that they continued to struggle with trying to
“fit” the entire lesson plan into classroom time allocated to math.


Teachers were nearly unanimous (96.5%) in agreeing that implementation of the
curriculum got easier as the year progressed. This opinion was further supported in
open-ended responses where one teacher summed up the comments this way,


I feel like I will be better prepared overall, just because I now know what comes
throughout the entire year. I will also prepare for the specific areas that were
challenging to this year’s group, and present it in a more guided way.


In an open-ended query, teachers were asked what they planned to do
differently when implementing
IL Illustrative Math in the 202324 school year. Out of
190 teacher responses, six teachers (3.2%) indicated they would make no changes, five
Imagine Learning Illustrative Math Fort Zumwalt 18
© Johns Hopkins University, 2023

(2.6%) answered “N/A,” and three (1.6%) indicated they did not know. The most
common response given from over 40% of the remaining respondents (n = 78, 41.1%)
was the intention to be better prepared/more organized in their implementation in the
next school year. Roughly half of these teachers (n = 40, 21.1%) specified that they
wanted to improve their organization when it came to teaching Centers and maintaining
Center materials. Teachers said they planned to prep their Centers early/over the
summer and others spoke of improving the way they prepped and stored items like
activity cards and games. The other main differences that teachers planned to make
were as follows:


Change in the timing/length of time of Centers (n = 42, 22.1%). Roughly half
of these teachers (n = 23, 12.1%) planned to use Centers more frequently
while roughly one-third (n = 12, 6.3%) intended to move Centers to the start
of the daily math lesson.

Improve time management/pacing during lessons (n = 30, 15.8%). Teachers
intended to be more cognizant of the time being spent on curricular activities
in order to “fit” everything in. Some (n = 10, 5.3%) proposed using timers to
assist this effort.


Related teacher comments included:


I feel more confident with the curriculum, so trying to experiment more
with the centers with student pairings, intervention groups, etc. is a goal
of mine. There have been things I've learned through this year about
teaching the center activities (inviting students to play against the
teacher, or in a central place in the room), structured classroom pairings
(A/B partnerships), and careful consideration of assessments. I would like
to try to plan misconceptions or missing links in prior learnings for
students who have not experienced ILIM.


[IL Illustrative Math] prep is way more work than [the] previous
curriculum. Prepping for centers and activities is very time consuming. It
would have been nice if the material was provided to you like our previous
curriculum. Other than the prep I have enjoyed ILIM. My students really
enjoyed it.


Next year, I expect to benefit from the students coming in having a year
of IM experience, which will allow me to implement better routines and
math practices from the very start of the year. I will have better pacing
and management when it comes to prioritizing certain activities and
ensuring that warm ups and the synthesis happens each day in a
meaningful way.

Imagine Learning Illustrative Math Fort Zumwalt 19
© Johns Hopkins University, 2023

Finally, smaller numbers of teachers (less than five) indicated that they would
make changes to other aspects of their implementation, such as increasing use of the
digital component of the curriculum, using the curriculum with greater consistency, and
providing occasional direct instruction.


Perceived Impact on Student Learning


Teachers reported on their perceived impact of
IL Illustrative Math on student
learners in terms of engagement and student achievement. Figure 4 displays these
findings.


Figure 4


Perceived Impact on Student Engagement


Note. + < 5%.

Largely, teachers agreed the curriculum motivated students to persist through
difficult content (85.7%), challenged students appropriately (92.2%), and led students
to engage in high-level discussions during math instruction (92.1%).


Teachers were presented with a set of three items relating to Imagine Learning
Illustrative Mathematics’ impact on student learning. Figure 5 details these findings.







+
+
+
6.9%
6.9%
10.8%
43.3%
36.0%
52.7%
48.8%
56.2%
33.0%
Students discussed mathematics at a high-level
during mathematics instruction
Students were appropriately challenged during
mathematics instruction
Students were motivated to persist through difficult
content
Perceived Impact on Student Learning:
Motivation, Challenge, & Discussion
Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree
Imagine Learning Illustrative Math Fort Zumwalt 20
© Johns Hopkins University, 2023

Figure 5


Perceived Impact of
IL Illustrative Math on Student Mathematics Learning

Note. + < 5%.

There was strong agreement among teachers that the curriculum positively
impacted student attitudes towards math, their ability to work in groups, and their
ability to problem solve mathematically, with roughly half of all teachers strongly
agreeing with these statements. Tied to these findings were responses to a second
open-ended query which asked teachers to describe how the
IL Illustrative Math
curriculum differed from other math curricula to a teacher who was new to IL

Illustrative Math
or to teaching math. The most commonly cited differences referenced
by 183 respondents were as follows:


It is a student-led, hands-on/problem-based curriculum (n = 54, 28.7%)
The curriculum provides students with a deeper understanding of math (n =
43, 22.9%)

The curriculum incorporates extensive group and partner work (n = 40,
21.3%)


Nearly 30% of teachers spoke of
IL Illustrative Maths hands-on format for
student learning. One teacher related, “This curriculum allows for students to discover
their own learning with the teacher there as a guide. It is extremely engaging for
students and the activities are very kid-friendly,” and another added, “Much, much
more hands on for the kids and a lot more fun.” Speaking on the implementation from a
teacher’s point of view a teacher noted that, “Moving to a new way of teach[sic] is
always challenging and the problem-based approach is unfamiliar to many. However,

+
+
+
8.4%
7.4%
12.8%
45.3%
40.9%
36.5%
44.8%
50.2%
45.8%
The curriculum improved students' ability to problem
solve mathematically
The curriculum improved students' ability to work in
groups
The curriculum improved students' attitudes towards
mathematics
Program Impact on Student Learning:
Attitude, Group Work, & Problem Solving
Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree
Imagine Learning Illustrative Math Fort Zumwalt 21
© Johns Hopkins University, 2023

we were able to see great student success with this program.” Another teacher spoke
to the challenge of implementing a new curriculum saying:


My workflow as a teacher is very different using IM curriculum resources.
Instead of spending lots of hours creating or curating lesson activities,
practice problems, and assessments these resources are already ready to
use. But since I didn't need to do that work, I had to take more time
working through the materials "like a student" to better understand the
progressions of concepts and the intention behind some activities (things
that I could just skim over in the past when I had done that work on the
front end by creating or finding/adjusting over resources).


Teachers who commented on the curriculum’s ability to deepen a student’s
understanding of math provided feedback with one saying, “This math curriculum
creates mathematical thinkers inside of your classroom. It will get students talking and
really thinking about their math, rather than just producers,” and another adding:


This curriculum is different in a way that it "forces" students to think
about mathematical situations in a deeper level of thinking ... I also think
this curriculum ties many of the mathematical problems to real work
situations which is HUGE for students.


Teachers who noted the curriculum’s high level of collaborative partner and
group work viewed this as a curriculum strength. Teachers described a positive impact
on student learning gained through students having opportunities to practice math
language and problem-solving with others and being able to share their ideas/thinking.
One teacher spoke to this point saying:


The dynamics that come from the problem solving, inquiry model and
discussion supports has opened math thinking up for all learners. Students
are gaining entry points to the learning, questioning peers for clarity or
understanding, and showing me things I never would have thought about
before. Trust in the depth and the challenge...the students will rise to it!


Smaller numbers of teachers identified several additional differences of
IL Illustrative
Math
to other math curricula:

IL Illustrative Math is engaging to students (n = 23, 12.2%)
Implementation requires additional prep time/organization (n = 21, 11.2%)

The curriculum is inclusive of all ability levels/allows for differentiation (n =
21, 11.2%)

Teachers find it user-friendly/easy to follow (n = 18, 9.6%)

Imagine Learning Illustrative Math Fort Zumwalt 22
© Johns Hopkins University, 2023

Overall Perceptions


Teachers were asked to provide their overall perceptions of the curriculum in
terms of whether respondents would recommend the program to others, if it had
provided a time-saving benefit versus previously used curricula, and whether it had
been more effective than a previously used curriculum (see Figure 6).


Figure 6

Overall Perceptions Regarding IL Illustrative Math



Over 85% of participants agreed that they would recommend the curriculum to
another teacher who is not using the curriculum. Over half (59.6%) of the participants
indicated that IL Illustrative Math had provided a time savings benefit. However, over
three-quarters of teachers (78.3%) agreed that the program had been more effective
than the previously used curriculum.


The fourth and last open-ended query asked teachers what advice they would
give about implementing the curriculum effectively to another teacher who was new to
IL Illustrative Math. Just as in the previous query, the most common advice offered in
the 191 responses was that implementers should plan ahead/be organized (n = 85,
44.3%). Suggestions for this included doing prep work prior to the week of teaching,
setting a routine for instruction, and organizing Centers. One teacher stated, “If you are
prepared and know your content, your students will rise to the occasion,” and another
added:


Although it is a lot of prep, in the end, it was worth it! The knowledge and
understanding that my students have about math is so much deeper than

5.4%
14.8%
5.4%
16.3%
25.6%
7.9%
40.9%
37.9%
36.5%
37.4%
21.7%
50.2%
Imagine Learning Illustrative Mathematics has been
more effective than previously used curriculum.
Imagine Learning Illustrative Mathematics has provided
a time savings benefit in comparison to previously used
curriculum.
I would recommend Imagine Learning Illustrative
Mathematics to another teacher who is not using the
curriculum.
Overall Perceptions
Strongly disagree Somewhat disagree Somewhat agree Strongly agree
Imagine Learning Illustrative Math Fort Zumwalt 23
© Johns Hopkins University, 2023

with prior curriculums. It was a mindset change on my behalf and did take
some adjusting on my end. Allow students to share, explain, and have
those conversations amongst peers. I would say let it get messy and trust
the process!


The advice to “be patient” and “take your time” was the next most frequently
named guidance noted by teachers (n = 50, 26.0%). One teacher offered reassurance
saying, Stick with it! It can be overwhelming (a lot of prep) at the beginning, but once
you get the hang of it and get a good system down it's definitely a great program!” and
another noted, “Take things day by day! It will get easier as you become more
comfortable with the curriculum.” Other recommendations to new implementers
included being thoughtful about time management (n = 20, 10.4%), making use of
resources such as notes and slides (n = 18, 9.4%), and the recommendation that
nothing from the curriculum be “skipped” or “left out” (n = 10, 5.2%). As one teacher
noted, “Follow and trust the process. The warm-ups lead to the activities. The activities
lead to the cool-down. They all work together.” Finally, smaller numbers of teachers
advised new implementers to remain flexible, to let their students talk through
problems together, and to work with their teaching team for support.


Taken together, these findings underscore the teacher perceptions that
IL
Illustrative Math
positively impacted student learning both in terms of student
engagement and achievement in mathematics instruction. Teachers also strongly
agreed that implementation of the curriculum became easier as the school year
progressed.


Discussion

The current study was a mixed-methods study designed to provide efficacy and
teacher satisfaction evidence for the IL Illustrative Math program. Impacts on student
mathematics achievement were determined for Grades 38 students in the Fort
Zumwalt School District by comparing treatment students who participated in IL
Illustrative Math with those who did not. This report includes findings from student
achievement, as well as teacher perceptions obtained through a questionnaire
administered to teachers who used IL Illustrative Math.


Elementary school results showed that Grades 4 and 5 students who used IL
Illustrative Math significantly outgained comparison students on the Spring 2023 MAP
mathematics assessment. IL Illustrative Math students averaged nearly 11-point larger
gains from spring 2022 to spring 2023. No significant impact was found for Grade 3
students, but effect sizes of 0.28 SDs for Grades 4 and 5 students and 0.57 SDs for
Grade 3 students, indicated medium to large practical program impacts. No significant
impacts of IL Illustrative Math on GCAS mathematics scores were observed for
elementary students. Subgroup analyses showed a significant positive impact of IL
Illustrative Math on MAP mathematics scores for special education students, with
Imagine Learning Illustrative Math Fort Zumwalt 24
© Johns Hopkins University, 2023

treatment special education students outscoring comparison special education students
by more than 13 points.


Results for middle school (Grades 7 and 8) showed that students who used IL
Illustrative Math significantly outgained comparison students on the Winter 2023 GCAS
mathematics assessments. In relation to comparison students, IL Illustrative Math
students averaged nearly 16-point larger gains from BOY to MOY in the 2022-23 school
year. The effect size of 0.16 SDs shows both practical and statistical significance of
these impacts. No statistically significant program impacts on MAP mathematics scores
were found in these analyses; however, a significant positive program impact on
Geometry MAP subscale scores was observed. In addition, a significant positive impact
of IL Illustrative Math on GCAS mathematics scores was observed for special education
students, with treatment special education students outscoring comparison special
education students by more than 26 points. Taken together, these analyses provide
efficacy evidence that may reach ESSA Tier 2 (“Moderate”) levels for elementary school
grades.


Teacher perceptions of IL Illustrative Math were generally positive with most
teachers (87%) agreeing that they would recommend the curriculum to others.
Teachers provided particularly positive feedback regarding the curriculum appropriately
challenging students during math instruction, improving students’ ability to work in
groups, and enabling high-level student discussion of mathematics during instruction. In
describing how IL Illustrative Math differed from other math curricula, teachers
highlighted that it was student-led, provided students with a deeper understanding of
math, and that it incorporated extensive group/partner work into math instruction.
Additionally, there was strong agreement amongst teachers that the curriculum helped
students persist through difficult content and improved their ability to problem solve
mathematically, and over 80% of teachers agreed that the curriculum improved
students’ attitudes towards mathematics.


Learning to implement the curriculum within the time limits of the mathematics
block appeared to be challenging for teachers, especially early in the school year. Many
teachers (70.4%) indicated that they had deviated from the typical lesson flow, either
by doing less due to a lack of time, or by re-ordering the timing of Centers. In open-
ended responses, teachers stated that to combat this challenge they intended to be
better prepared/organized in the next school year, especially when it came to Centers.


Teachers agreed that they had received adequate support both from their district
and school administrations during implementation. They also agreed that they had
received sufficient professional development to successfully implement the various
curricular components of IL Illustrative Math including the digital tools and resources
associated with the curriculum, although agreement was somewhat lower for the digital
piece. Notably, over 75% of teachers agreed that IL Illustrative Math was more
effective than the previously used curriculum, while being slightly less likely to perceive
Imagine Learning Illustrative Math Fort Zumwalt 25
© Johns Hopkins University, 2023

it as providing a time savings benefit (40% disagreed that it saved time). Finally, nearly
all teachers (97%) agreed that implementation got easier as the school year
progressed. In response to an open-ended query, roughly 45% of respondents stated
that they would advise new implementers of the curriculum to plan ahead and be
organized in order to be successful.


Conclusions


In considering overall conclusions from this study, it is important to consider that
IL Illustrative Math is a relatively new program that asks teachers to make considerable
changes to their classroom processes and structures and this study captures a first-year
implementation of IL Illustrative Math in FZSD. Questionnaire data indicates that
teachers, who generally perceived the program very positively, took considerable time
becoming comfortable with program implementation. Another necessary consideration
is the nature of how the GCAS assessment was administered in middle grades. While
the FZSD BOY and MOY GCAS assessments function as typical progress monitoring
assessments, EOY GCAS assessments are more summative in nature and, thus, function
somewhat differently than BOY and MOY assessments, and scores also should be
interpreted differently. These intra-year assessment differences may help to explain the
different patterns of results between elementary students, who only took the GCAS at
BOY and EOY, and middle school students, who took the GCAS at all three timepoints.
The significant positive findings for elementary IL Illustrative Math students on the MAP
mathematics assessment provide additional evidence of the efficacy of IL Illustrative
Math in the elementary grades. It is important to note that the GCAS assessments are
originally designed as a benchmarking assessment to be administered at three time
points throughout the year. As such, this adjustment in FZSD justifies the analysis in
middle grades focusing on BOY to MOY results as this truly captures students’ progress
since the middle grades EOY assessment was more summative in nature.


In all, the results of this evaluation showed generally positive findings regarding
both student achievement impacts and teacher program perceptions relating to IL
Illustrative Math. At the elementary and middle school levels, statistically significant
overall program effects (and associated moderately strong effect sizes) on MAP and
GCAS, respectively, appear to provide “moderate” evidence support (Tier 2) for both
ESSA and the What Works Clearinghouse (meets standards with reservations). One
limitation of this evaluation was that program usage data were not available, as IL
Illustrative Math contains both digital and print components, thus prohibiting the ability
to accurately calculate the total time that students engaged with the curriculum and
program components. Program usage data may help to examine potential associations
between program dosage and achievement gains, and other similar finer-grained
analyses of observed achievement gains. Further evaluation is encouraged in
subsequent years to further examine potential program impacts of IL Illustrative Math
on mathematics achievement, especially as teachers gain greater experience and
comfort with the program.
Imagine Learning Illustrative Math Fort Zumwalt 26
© Johns Hopkins University, 2023


Appendix A: Teacher Questionnaire

Please indicate your school.

Dubray Middle School

North Middle School

South Middle School

West Middle School

Dardenne Elementary School

Emge Elementary School

Flint Hill Elementary School

Forest Park Elementary School

Hawthorn Elementary School

JL Mudd Elementary

Lewis & Clark Elementary School

Mid Rivers Elementary School

Mount Hope Elementary School

Ostmann Elementary School

Pheasant Point Elementary School

Progress South Elementary School

Rock Creek Elementary School

St. Peters Elementary School

Twin Chimneys Elementary School

Westhoff Elementary School



What is your primary role?

Classroom teacher

Interventionist

Instructional Aid/Paraprofessional

Coach

Other, please specify: __________________________________________________

Imagine Learning Illustrative Math Fort Zumwalt 27
© Johns Hopkins University, 2023


What grade(s) do you teach? (Check all that apply.)

Kindergarten

First Grade

Second Grade

Third Grade

Fourth Grade

Fifth Grade

Sixth Grade

Seventh Grade

Eighth Grade

None of the above


Have you used ILIM prior to the 2022-2023 school year?

Yes

No


When teaching math using ILIM, how often do you do the following (either on paper or on the
digital platform)?

0 times per
week

1-2 times per
week

3-4 times per
week

5 days per
week
N/A
Lesson
Warm-Up

Lesson Cool
Down

Lesson
Synthesis

Centers (K-5
only)


Imagine Learning Illustrative Math Fort Zumwalt 28
© Johns Hopkins University, 2023

Please rate the extent to which you agree or disagree with each of the following statements.
Imagine Learning Illustrative Math Fort Zumwalt 29
© Johns Hopkins University, 2023

Strongly agree Somewhat
agree

Somewhat
disagree

Strongly
disagree

Students were
motivated to
persist through
difficult content.


Students were
appropriately
challenged
during
mathematics
instruction.


Students
discussed
mathematics at
a high-level
during
mathematics
instruction.


The curriculum
improved
students’
attitudes
towards
mathematics.


The curriculum
improved
students’ ability
to work in
groups.


The curriculum
improved
students’ ability
to problem solve
mathematically.


Implementing
the curriculum
got easier as the
year progressed.

Imagine Learning Illustrative Math Fort Zumwalt 30
© Johns Hopkins University, 2023

I received
sufficient
professional
development to
successfully
implement and
utilize the digital
tools and
resources
available in the
curriculum.


I received
sufficient
professional
development to
successfully
implement the
various
curriculum
components
(e.g., Warm-Up,
Activities, Cool-
downs, etc.)


I received
adequate
ongoing support
from district
administration.


I received
adequate
ongoing support
from school
administration
(e.g., principals,
coaches).


I would
recommend ILIM
to another
teacher who is
not using the
curriculum.

Imagine Learning Illustrative Math Fort Zumwalt 31
© Johns Hopkins University, 2023

ILIM has
provided a time
savings benefit
in comparison to
previously used
curriculum.


ILIM has been
more effective
than previously
used curriculum.




A typical ILIM lesson starts with a warm up, includes at least one activity, and ends with a
synthesis, cool down, and, if you are in Grades K5, centers. If you deviated from this typical
flow for a lesson, describe what you did differently and why.


What would you say about how this curriculum is different from other math curricula to
another teacher who was new to ILIM? Someone who is new to teaching math?


As you look toward next year, what will you do differently when you utilize ILIM in your
classroom?


What advice would you give about implementing the curriculum effectively to another teacher
who was new to ILIM?

















Imagine Learning Illustrative Math Fort Zumwalt 32
© Johns Hopkins University, 2023

Appendix B: Baseline Equivalence Tables

Table B1


Baseline Equivalence, 2023 BOY GCAS Scores, Grades 3-5


Overall
Mean

Imagine
Mean

(SD)

Control
Mean

(SD)

Adjusted
T v C
Difference

Pooled
Unadjusted
SD

Stan.
Mean
Diff.

Grade 3
721.27 720.65
(73.36)

721.87
(70.50)

-1.21
71.92 -0.02
Grade 4
808.51 809.12
(68.39)

808.17
(69.61

0.95
69.18 0.01
Grade 5
898.52 892.22
(77.74)

904.61
(79.37)

-12.39
78.57 -0.16
All Students
810.09 807.76
(104.21)

811.95
(102.11)

-4.19
103.05 -0.04

Table B2


Baseline Equivalence, 2023 BOY GCAS Scores, Grades 7-8


Overall
Mean

Imagine
Mean

(SD)

Control
Mean

(SD)

Adjusted
T v C
Difference

Pooled
Unadjusted
SD

Stan.
Mean
Diff.

Grade 7
1160.88 1164.04
(77.17)

1145.01
(76.05)

19.024
76.99 0.25
Grade 8
1219.09 1216.64
(48.68)

1223.98
(45.60)

-7.34
47.68 -0.15
All Students
1185.36 1183.36
(72.66)

1191.89
(71.29)

-8.54
72.34 -0.12












Imagine Learning Illustrative Math Fort Zumwalt 33
© Johns Hopkins University, 2023

Appendix C: Descriptive Achievement Tables

Table C1


Elementary GCAS Mathematics Scores


BOY MOY N Growth
Grade 3

IL Illustrative Math
720.65 (73.36) 820.45 (98.62) 504 99.80
Comparison
721.87 (70.50) 820.70 (99.46) 519 98.83
Grade 4

IL Illustrative Math
809.12 (68.39) 944.02 (97.46) 364 134.90
Comparison
808.17 (69.61) 956.27 (105.13) 673 148.10
Grade 5

IL Illustrative Math
892.22 (77.74) 1021.41 (99.17) 514 129.19
Comparison
904.61 (79.37) 1036.75 (98.62) 532 132.14
Note. SD in parentheses.


Table C2


Middle School GCAS Mathematics Scores


BOY MOY N Growth
Grade 6

IL Illustrative Math
1069.67 (75.44) 1153.75 (80.71) 1,068 86.08
Comparison
972.36 (66.54) 1007.64 (81.47) 47 35.28
Grade 7

IL Illustrative Math
1164.04 (77.17) 1208.46 (85.60) 906 44.42
Comparison
1145.01 (76.05) 1173.16 (85.27) 180 28.15
Grade 8

IL Illustrative Math
1216.64 (48.68) 1291.31 (92.22) 526 74.67
Comparison
1223.98 (45.60) 1293.45 (87.45) 263 69.47
Note. SD in parentheses.


Table C3


Elementary MAP Mathematics Scores


BOY MOY N Growth
Grade 4

IL Illustrative Math
367.53 (37.58) 396.29 (41.98) 347 28.76
Comparison
357.48 (39.58) 396.44 (39.90) 644 38.96
Grade 5

IL Illustrative Math
388.57 (38.00) 409.71 (35.92) 494 31.14
Imagine Learning Illustrative Math Fort Zumwalt 34
© Johns Hopkins University, 2023

Comparison
397.93 (38.32) 408.09 (34.75) 509 20.16 Note. SD in parentheses.

Table C4


Middle School MAP Mathematics Scores


BOY MOY N Growth
Grade 6

IL Illustrative Math
413.28 (30.44) 425.46 (30.33) 1,037 12.18
Comparison
334.98 (33.63) 361.98 (35.33) 47 27.00
Grade 7

IL Illustrative Math
423.68 (30.26) 441.71 (38.40) 892 18.03
Comparison
409.32 (39.44) 433.76 (44.29) 174 24.44
Grade 8

IL Illustrative Math
428.23 (33.93) 461.61 (46.72) 521 33.38
Comparison
426.83 (32.46) 466.39 (39.60) 254 39.56
Note. SD in parentheses.



























Imagine Learning Illustrative Math Fort Zumwalt 35
© Johns Hopkins University, 2023

Appendix D: MAP Subscale Results

Table D1


MAP Mathematics Subscale Regression Results, Grades 45


Subscale
Estimate Standard Error p value
Base Ten Numbers
-0.086 6.094 .989
Fractions
13.109* 5.180 .011
Algebraic Thinking
12.557* 5.968 .035
Geometry
16.280** 5.192 .002
Note. * p < .05; ** p < .01.


Table D2


MAP Mathematics Subscale Regression Results, Grade 3


Subscale
Estimate Standard Error p value
Base Ten Numbers
36.645 30.469 .229
Fractions
43.836 27.158 .107
Algebraic Thinking
8.246 18.580 .657
Geometry
32.571 21.059 .122

Table D3


MAP Mathematics Subscale Regression Results, Grade 78


Subscale
Estimate Standard Error p value
Rations/Proportions
1.010 2.743 .713
Number Sense
-0.895 3.786 .813
Equations/Inequalities
-7.278* 3.351 .030
Geometry/Measurement
6.969** 2.537 .006
Note. * p < .05; ** p < .01.










Imagine Learning Illustrative Math Fort Zumwalt 36
© Johns Hopkins University, 2023

Appendix E: Subgroup Regression Tables Elementary

All regression models controlled for prior mathematics achievement, grade,
FARMS, and SPED status, as well as school effects. In addition, all variables were grand
mean centered to facilitate interpretation of the intercept. Student and classroom sizes
were identical to those outlined in previous regression tables. Note that the treatment
effect for each subgroup was calculated by adding the overall treatment effect and the
treatment interaction term.


Table E1


GCAS Mathematics Regression Results with SPED Interaction


Estimate Standard Error p value
IL Illustrative Math
-6.835 5.385 .204
Illustrative*SPED
15.901* 7.502 .034
SPED
-58.019*** 5.424 <.001
Constant
935.651*** 3.424 <.001
Note. * p < .05; *** p < .01.


Table E2


GCAS Mathematics Regression Results with FARMS Interaction


Estimate Standard Error p value
IL Illustrative Math
-3.619 5.364 .500
Illustrative*FARMS
-0.497 6.948 .943
FARMS
-21.745*** 4.682 <.001
Constant
935.411*** 3.424 <.001
Note. *** p < .001.


Table E3


GCAS Mathematics Regression Results with ELL Interaction


Estimate Standard Error p value
IL Illustrative Math
-3.434 5.226 .511
Illustrative*ELL
-4.934 11.841 .677
ELL
4.764 9.109 .601
Constant
935.409*** 3.423 <.001
Note. *** p < .001.


Table E4