(1 of 12)

Current View
Impact Evaluation of Imagine
Language & Literacy in a Large
Southeast School District
Kayla Freeman, Natasha Wilson, and Drew Berrett
March 2024
IntroductionAccording to the most recent National Assessment of Educational Progress (NAEP) Reading Report
Card, elementary English language learners scored significantly lower on the 2022 reading
assessment than non-English language learners (U.S. DOE, n.d.). Although it is critical to improve
reading scores across all student subgroups, it is imperative to close this gap for English language
learners as they make up about 10% of America’s student population (NCES, 2023).
Digital learning tools can be a valuable way to support all students in developing English language
proficiency (Rahmati et al., 2021). Imagine Language & Literacy by Imagine Learning is a
digital supplemental English language solution designed to personalize learning for students
through direct, explicit, and systematic instruction and practice that ensures students learn
critical skills in four language domains. The program provides personalized learning pathways
for each student that adapt automatically to maximize engagement and progress. As such,
students who utilize Imagine Language & Literacy are expected to improve and accelerate
their English language proficiency.
In partnership with a southeastern school district, Imagine Learning conducted a study designed
to evaluate the efficacy of Imagine Language & Literacy. The primary research question was:
how does use of Imagine Language & Literacy impact Grades 1–5 English language learners’
performance on a test of English language proficiency? Reported study results demonstrate
how the program positively impacted students’ literacy proficiency by comparing the performance
of Imagine Language & Literacy students to a highly similar group of students who did not use
the Imagine Language & Literacy program.
Methods
POPULATION
During the 2022–2023 school year, Imagine Language & Literacy was made available to
Grades 1–5 English language learners in a large Southeast school district. A total of 46 schools
enrolled students who used the Imagine Language & Literacy program for more than zero
minutes during the school year. In these schools, Imagine Language & Literacy was used at
the discretion of teachers and families if it was deemed valuable to support the learning of an
individual student. A total of 2,757 students in those schools used the program and 901 students
did not. In addition, data were collected for 218 students from six schools that had no Imagine
Language & Literacy usage. Ultimately, a total of 1,119 students did not use the Imagine Language
& Literacy program while a total of 2,757 students were categorized as program users.
1Impact Evaluation of Imagine Language & Literacy in a Large Southeast School District
RESEARCH DESIGN
This study was conducted retrospectively using data from the 2022–2023 school year. It
evaluated the difference in English language acquisition between treatment (users of Imagine
Language & Literacy) and control (non-users of Imagine Language & Literacy) students. The
treatment group was comprised of students who logged any usage in the Imagine Language
& Literacy program during the 2022–2023 school year, whereas the control group included all
students who did not. Assignment to the treatment and control groups was not random, so this
study is a quasi-experimental design, and statistical procedures were used to ensure baseline
equivalence of the treatment and control samples. Because use of Imagine Language &
Literacy was determined for individual students rather than entire classrooms or schools,
statistical corrections for clustering were not required.
MEASURES
Multiple data sources were compiled to describe students, their performance, and their work in
Imagine Language & Literacy. Student English language proficiency outcomes were determined
using a standardized progress monitoring assessment. Student demographic data were collected
to provide additional information on student characteristics that may impact measures of
learning outcomes. Data from the Imagine Language & Literacy program were incorporated
to evaluate student engagement. These data sources are reviewed in more detail below.
English Language Proficiency. Students’ English language proficiency was determined using
Louisiana’s English Language Proficiency Test (ELPT). ELPT scores were obtained for students
who completed the assessment in 2022 and 2023. The ELPT is administered each year from
mid-February to mid-March. Scores from 2022 were used to establish baseline equivalence
between study groups, and 2023 scores were used to estimate the effect of Imagine Language
& Literacy on English language proficiency. Students are not assigned an overall score on the
ELPT but are assigned subscale scores. The district in the study is rated based on their students’
performance on growth trajectories assigned by the state for the subscales. As such, student
outcomes on the ELPT were categorized as a dichotomous outcome (1 = met/exceeded
growth trajectory or 0 = did not meet/exceed growth trajectory).
Student Demographics. Information was collected on individual student demographic
characteristics including grade level, gender, race/ethnicity, special education status, free
or reduced-price lunch status, and years in the English-learner program.
Imagine Language & Literacy Usage. Program usage data were obtained to determine
students’ engagement and progress in Imagine Language & Literacy. These data included the
total minutes students spent in the program and the number of lessons students passed.
2Impact Evaluation of Imagine Language & Literacy in a Large Southeast School District
ANALYTICAL SAMPLE
To ensure that the baseline characteristics of treatment and control students used in analyses
were comparable, propensity score matching was used to create a statistically equivalent
analytical sample.1 Control students were matched to treatment students based on their
2022 ELPT reading, writing, listening, and speaking subdomain scores and all demographic
information available: grade level, race/ethnicity, gender, special education status, free or
reduced-price lunch status, and years in the English-learner program. Exact matching was
used for grade level and the number of years in the English-learner program. Grade level was
chosen as prior analyses on Imagine Language & Literacy have indicated this factor is likely to
have a large impact on English language proficiency. Number of years in English-learner program
was chosen as this value is considered when determining a student’s growth expectation and
likely also has a large impact on English language proficiency. The resulting analytical sample
included 628 users of Imagine Language & Literacy and 628 non-users. Table 1 below describes
the characteristics of the sample. To ensure that the results are not sensitive to the final analytic
sample chosen, a second analytic sample was made that required exact matches on all available
demographic characteristics. Further details of this sample and the resulting analyses can be
found in Appendix A.
ANALYTICAL APPROACH
In Louisiana, public school districts are rated on their students’ performance based on growth
trajectories assigned by the state. Students fall into one of four categories: “ELPT level is the
same or lower than prior year,” “ELPT level is at least one level higher than prior year,” “ELPT level
meets trajectory,” and “ELPT level exceeds trajectory” (Louisiana Believes, 2022). The district was
interested in whether use of the literacy intervention is associated with meeting or exceeding
the growth trajectory. To answer this question, logistic regression was used to compute the
odds that Grades 1–5 students met or exceeded their growth trajectory using a dichotomous
outcome variable, 1=met/exceeded and 0 = did not meet/exceed. These odds were then
compared between the Imagine Language & Literacy user students and the non-user students,
controlling for 2022 ELPT scores and demographic variables. An indicator of whether a student
was a control or treatment student was included in the regression as the primary predictor
variable. Using logistic regression after propensity score matching ensured that any remaining
differences in the underlying treatment and control samples were controlled for by the regression
model, effectively isolating the impact of Imagine Language & Literacy.
1 One-to-one nearest neighbor propensity score matching without replacement, with a caliper set to 0.009, was
executed using the matchit function in R’s MatchIt package.
3Impact Evaluation of Imagine Language & Literacy in a Large Southeast School District
Table 1. Baseline Equivalence.
Control (n = 628) Treatment (n = 628) p-value SMD
Sp. 22 Reading Score
(mean/SD)
501.57 (62.33) 499.42 (62.86) 0.544 0.034
Sp. 22 Writing Score
(mean/SD)
491.40 (67.81) 490.98 (68.14) 0.913 0.006
Sp. 22 Listening Score
(mean/SD)
510.93 (70.79) 505.72 (70.78) 0.193 0.073
Sp. 22 Speaking Score
(mean/SD)
511.00 (87.26) 507.30 (83.30) 0.443 0.043
Grade Level 1.000 <0.001
Grade 1 (%) 178 (28.3) 178 (28.3)
Grade 2 (%) 120 (19.1) 120 (19.1)
Grade 3 (%) 112 (17.8) 112 (17.8)
Grade 4 (%) 122 (19.4) 122 (19.4)
Grade 5 (%) 96 (15.3) 96 (15.3)
Gender 0.309 0.061
Female (%) 292 (46.5) 311 (49.5)
Male (%) 336 (53.5) 317 (50.5)
Ethnicity 0.887 0.060
Asian (%) 19 (3.0) 20 (3.2)
Black (%) 15 (2.4) 15 (2.4)
Hispanic (%) 577 (91.9) 575 (91.6)
Other (%) 1 (0.2) 0 (0.0)
White (%) 16 (2.5) 18 (2.9)
SPED Classification 0.766 0.022
Not SPED (%) 570 (90.8) 574 (91.4)
SPED (%) 58 (9.2) 54 (8.6)
FRL Status 0.367 0.054
Free Lunch (%) 318 (50.6) 301 (47.9)
Paid Lunch (%) 310 (49.4) 327 (52.1)
Years in ELL program
(mean/SD)
3.39 (1.39) 3.39 (1.39) 1.000 <0.001
4Impact Evaluation of Imagine Language & Literacy in a Large Southeast School District
Results
IMAGINE LANGUAGE & LITERACY USAGE
Matched treatment students spent an average of 10.5 hours (with a median of 5.5 hours) in
Imagine Language & Literacy and passed an average of 12.5 lessons (with a median of 6).
See Figures 1 and 2 for a distribution of hours and lessons passed.
Figure 1. Distribution of Hours Spent in Imagine Language & Literacy by Grade.
50
40
30
20
10
1 2 543
50
40
30
20
10
1 2 543

Hours in Imagine Lanugage & Literacy
Grade
Figure 2. Distribution of Lessons Passed in Imagine Language & Literacy by Grade.
50
40
30
20
10
1 2 543
50
40
30
20
10
1 2 543
Lessons Passed Imagine Lanugage & Literacy
Grade
Note: Outliers that fall above 1.5 times the interquartile range are not included in this figure to ensure readability.
The global maximum hours spent in Imagine Language & Literacy is 77.1 hours. The global maximum lessons passed
in Imagine Language & Literacy is 116 lessons.
5Impact Evaluation of Imagine Language & Literacy in a Large Southeast School District
PROGRAM IMPACT ON STUDENT ACHIEVEMENT
In evaluating the final matched sample, 40.3% of non-users (n = 253) and 45.1% of users (n = 283)
met growth expectations. A logistic regression found a positive and statistically significant
relationship between use of Imagine Language & Literacy and meeting/exceeding the ELPT
growth trajectory, B = .262, SE = .132, Wald = 3.9, p =.047. The estimated odds ratio favored an
increase of 30% [Exp (B) = 1.300, 95% CI (1.003, 1.682)] for meeting/exceeding growth expectations
among students who used Imagine Language & Literacy. The Cox index effect size of Imagine
Language & Literacy is 0.16.2 Table 2 summarizes the results of the logistic regression.
Table 2. Overall Impact of Imagine Language & Literacy on ELPT.
Coefficients Estimate Std. Error p-value
Intercept -4.00 .782 <.0001
Sp. 22 Reading Score .005 .002 .033
Sp. 22 Writing Score .011 .002 <.0001
Sp. 22 Listening Score .001 .002 .794
Sp. 22 Speaking Score -.002 .001 .084
Grade 2 .492 .198 .013
Grade 3 .462 .253 .068
Grade 4 .110 .278 .693
Grade 5 .690 .325 .034
Black -.769 .621 .216
Hispanic -1.02 .413 .014
Other 10.75 324.74 .974
White -.673 .566 .235
SPED Classification -1.08 .301 .0003
FRL Status .285 .144 .047
Years in ELL -.846 .102 <.0001
Treatment Variable .262 .132 .047*
2 The Cox index effect size is calculated by dividing the logistic regression coefficient by 1.65, which follows What
Works Clearinghouse’s Procedures and Standards Handbook, Version 5.0.
6Impact Evaluation of Imagine Language & Literacy in a Large Southeast School District
ConclusionThis study provides evidence of the efficacy of Imagine Language & Literacy on student English
language achievement for English language learners in Grades 1–5 by comparing students who
used Imagine Language & Literacy with those who did not during the 2022–2023 school year.
Results show that the odds of meeting growth expectations were 1.30 times higher for Grades
1–5 Imagine Language & Literacy user students than for non-user students. This difference was
statistically significant. Thus, this study provides evidence that the use of Imagine Language &
Literacy supports English language learners’ English language achievement.
References
Louisiana Believes. (2022). Understanding the ELPT and EL Accountability [PowerPoint slides]. Louisiana Department
of Education. http://www.louisianabelieves.com/docs/default-source/assessment/understanding-elpt-and-el-
accountability-summit-2022.pptx
National Center for Education Statistics. (2023). English Learners in Public Schools. Condition of Education. U.S.
Department of Education, Institute of Education Sciences. Retrieved January 31, 2024, from https://nces.ed.gov/
programs/coe/indicator/cgf.
Rahmati, J., Izadpanah, S., & Shahnavaz, A. (2021). A meta-analysis on educational technology in English language
teaching. Language Testing in Asia, 11(7). https://link.springer.com/content/pdf/10.1186/s40468-021-00121-w.pdf
U.S. Department of Education (DOE). (n.d.) Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2022 Reading Assessment.
What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0.
U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and
Regional Assistance (NCEE).
7Impact Evaluation of Imagine Language & Literacy in a Large Southeast School District
Appendix ATo ensure that observed results were not sensitive to the matching process used to select
the analytical sample, a second analytical sample was created using a different procedure.
Control students were matched to treatment students based on their 2022 ELPT reading,
writing, listening, and speaking subdomain scores and all demographic information available:
grade level, race/ethnicity, gender, special education status, free or reduced-price lunch
status, and years in the English-learner program.3 Exact matching was used for all demographic
characteristics: grade level, race/ethnicity, gender, special education status, free or reduced-
price lunch status, and years in the English-learner program. Exact matching on subdomain
scores was not used because it did not result in a large enough sample size. The resulting
analytical sample included 690 users of Imagine Language & Literacy and 690 non-users.
Table A1 below describes the characteristics of the sample.
3 One-to-one nearest neighbor propensity score matching without replacement, with a caliper set to 0.110, was executed
using the matchit function in R’s MatchIt package. Caliper values are chosen to maximize sample size while ensuring
adequate baseline equivalence; positive but insignificant results are found when a caliper below .110 is used.
8Impact Evaluation of Imagine Language & Literacy in a Large Southeast School District
Table A1. Baseline Equivalence.
Control (n = 690) Treatment (n = 690) p-value SMD
Sp. 22 Reading Score
(mean/SD)
501.66 (61.46) 492.18 (61.62) 0.004 0.154
Sp. 22 Writing Score
(mean/SD)
493.09 (67.28) 484.83 (69.96) 0.026 0.120
Sp. 22 Listening Score
(mean/SD)
511.41 (70.03) 499.42 (72.60) 0.002 0.168
Sp. 22 Speaking Score
(mean/SD)
512.67 (86.45) 501.43 (88.07) 0.017 0.129
Grade Level 1.000 <0.001
Grade 1 (%) 171 (24.8) 171 (24.8)
Grade 2 (%) 121 (17.5) 121 (17.5)
Grade 3 (%) 123 (17.8) 123 (17.8)
Grade 4 (%) 157 (22.8) 157 (22.8)
Grade 5 (%) 118 (17.1) 118 (17.1)
Gender 1.000 <0.001
Female (%) 323 (46.8) 323 (46.8)
Male (%) 367 (53.2) 367 (53.2)
Ethnicity 1.000 <0.001
Asian (%) 8 (1.2) 8 (1.2)
Black (%) 4 (0.6) 4 (0.6)
Hispanic (%) 660 (95.7) 660 (95.7)
Other (%) 0 (0.0) 0 (0.0)
White (%) 18 (2.6) 18 (2.6)
SPED Classification 1.000 <0.001
Not SPED (%) 649 (94.1) 649 (94.1)
SPED (%) 41 (5.9) 41 (5.9)
FRL Status 1.000 <0.001
Free Lunch (%) 338 (49.0) 338 (49.0)
Paid Lunch (%) 352 (51.0) 352 (51.0)
Years in ELL program
(mean/SD)
3.48 (1.39) 3.48 (1.39) 1.000 <0.001
9Impact Evaluation of Imagine Language & Literacy in a Large Southeast School District
Table A2. Overall Impact of Imagine Language & Literacy on ELPT.
Coefficients Estimate Std. Error p-value
Intercept -6.17 .907 <.0001
Sp. 22 Reading Score .006 .002 .009
Sp. 22 Writing Score .010 .002 <.0001
Sp. 22 Listening Score -.002 .002 .435
Sp. 22 Speaking Score .001 .001 .343
Grade 2 .458 .201 .023
Grade 3 .654 .236 .006
Grade 4 .280 .264 .288
Grade 5 1.10 .300 .0002
Black .778 1.12 .488
Hispanic .298 .638 .641
White .335 .738 .650
SPED Classification -1.04 .342 .002
FRL Status .388 .140 .006
Years in ELL -.973 .097 <.0001
Treatment Variable .339 .127 .008*
In evaluating the final matched sample, 40.7% of non-users (n = 281) and 44.1% of users (n = 304)
met growth expectations. A logistic regression found a positive and statistically significant
relationship between use of Imagine Language & Literacy and meeting/exceeding the ELPT
growth trajectory, B = .339, SE = .127, Wald = 7.1, p =.008. The estimated odds ratio favored an
increase of 40% [Exp (B) = 1.403, 95% CI (1.094, 1.800)] for meeting/exceeding growth expectations
among students who used Imagine Language & Literacy. The Cox index effect size of Imagine
Language & Literacy is 0.21.4 Table A2 summarizes the results of the logistic regression.
4 The Cox index effect size is calculated by dividing the logistic regression coefficient by 1.65, which follows What
Works Clearinghouse’s Procedures and Standards Handbook, Version 5.0.
10Impact Evaluation of Imagine Language & Literacy in a Large Southeast School District
imaginelearning.com/language-and-literacy
877-725-4257 • solutions@imaginelearning.com 1304749226 2403