Impact Evaluation of Imagine
Language & Literacy in a Florida
Public School District
Kayla Freeman, Natasha Wilson, Drew Berrett
May 2024
Language & Literacy in a Florida
Public School District
Kayla Freeman, Natasha Wilson, Drew Berrett
May 2024
IntroductionProviding effective strategies and tools for English language learners (ELLs), who represent about
10% of America’s student population (NCES, 2023), to achieve English language proficiency
is imperative. In fact, the 2022 National Assessment of Educational Progress (NAEP) Reading
Report Card indicates that elementary ELLs scored statistically significantly lower on the 2022
reading assessment than their non-English language learner peers (U.S. DOE, n.d.). Digital
learning tools are anticipated to be an effective way to support all students, especially ELLs, in
developing English language proficiency (Rahmati et al., 2021).
Technology can support ELLs develop English language proficiency in several ways. First, studies
have indicated that educational technology provides ELLs with opportunities to learn content
that interests them (Pourhossein Gilakjani, 2017; Solanki & Shyamleel, 2012) and provides them
with opportunities to develop autonomy in their learning (Pourhossein Gilakjani & Sabouri, 2014).
Further, educational technology can provide ELLs with opportunities to learn with support in
their native language (Park & Son, 2009). Finally, many educational technology platforms can
adapt to the learner’s initial and developing ability, providing a personalized and adaptive
learning opportunity that can support language acquisition (Faria et al., 2019).
Imagine Language & Literacy by Imagine Learning is a digital supplemental English language
solution designed to personalize learning for students through direct, explicit, and systematic
instruction and practice that ensures students learn critical skills in four language domains.
The program provides personalized learning pathways for each student that adapt automatically
to maximize engagement and progress. As such, students who utilize Imagine Language &
Literacy are expected to improve and accelerate their English language proficiency.
Methods
RESEARCH DESIGN
Imagine Learning partnered with a public school district in Florida to evaluate how Imagine
Language & Literacy impacted the success of its ELL students. During the 2022–2023 school
year, Imagine Language & Literacy was made available to ELL students in Grade 1 through
Grade 5 and was used at teachers’ discretion. In many cases, it was implemented in the
classroom or at home if a teacher deemed it valuable to support the learning of an individual
student outside of the classroom. ELL students who did not use Imagine Language & Literacy
were instead supported through the use of an array of district-provided resources including,
at times, other digital literacy programs.
1Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
10% of America’s student population (NCES, 2023), to achieve English language proficiency
is imperative. In fact, the 2022 National Assessment of Educational Progress (NAEP) Reading
Report Card indicates that elementary ELLs scored statistically significantly lower on the 2022
reading assessment than their non-English language learner peers (U.S. DOE, n.d.). Digital
learning tools are anticipated to be an effective way to support all students, especially ELLs, in
developing English language proficiency (Rahmati et al., 2021).
Technology can support ELLs develop English language proficiency in several ways. First, studies
have indicated that educational technology provides ELLs with opportunities to learn content
that interests them (Pourhossein Gilakjani, 2017; Solanki & Shyamleel, 2012) and provides them
with opportunities to develop autonomy in their learning (Pourhossein Gilakjani & Sabouri, 2014).
Further, educational technology can provide ELLs with opportunities to learn with support in
their native language (Park & Son, 2009). Finally, many educational technology platforms can
adapt to the learner’s initial and developing ability, providing a personalized and adaptive
learning opportunity that can support language acquisition (Faria et al., 2019).
Imagine Language & Literacy by Imagine Learning is a digital supplemental English language
solution designed to personalize learning for students through direct, explicit, and systematic
instruction and practice that ensures students learn critical skills in four language domains.
The program provides personalized learning pathways for each student that adapt automatically
to maximize engagement and progress. As such, students who utilize Imagine Language &
Literacy are expected to improve and accelerate their English language proficiency.
Methods
RESEARCH DESIGN
Imagine Learning partnered with a public school district in Florida to evaluate how Imagine
Language & Literacy impacted the success of its ELL students. During the 2022–2023 school
year, Imagine Language & Literacy was made available to ELL students in Grade 1 through
Grade 5 and was used at teachers’ discretion. In many cases, it was implemented in the
classroom or at home if a teacher deemed it valuable to support the learning of an individual
student outside of the classroom. ELL students who did not use Imagine Language & Literacy
were instead supported through the use of an array of district-provided resources including,
at times, other digital literacy programs.
1Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
This study was conducted retrospectively using data from the 2022–2023 school year to evaluate
the difference in literacy achievement between treatment and control students. The treatment
group was comprised of all ELL students who logged any usage in the Imagine Language &
Literacy program during the 2022–2023 school year, while the control group included all ELL
students who did not. This study is a quasi-experimental design as assignment to the treatment
and control groups was not random. Statistical procedures were used to ensure baseline
equivalence of the treatment and control samples.
MEASURES
Multiple data sources were compiled to describe students, their performance, and their work
in Imagine Language & Literacy. Student literacy proficiency outcomes were determined using
two standardized assessments. Student demographic data were collected to provide additional
information on student characteristics that may impact measures of learning outcomes. Data from
the Imagine Language & Literacy program were incorporated to evaluate student engagement
in Imagine Language & Literacy. These data sources are reviewed in more detail below.
Literacy Proficiency. Students’ English literacy proficiency was determined using the WIDA
ACCESS 2.0 (WIDA ACCESS) assessment for students in Grades 1 through 5 and the Florida
Assessment of Student Thinking (FAST) for students in Grades 4 and 5.
● WIDA ACCESS overall scale scores were obtained for students in 2022 and 2023. In Florida,
WIDA ACCESS is administered between January and March of each school year. Scores from
2022 were used to establish baseline equivalence between study groups, and 2023 scores
were used to estimate the effect of Imagine Language & Literacy on literacy proficiency.
● FAST scaled scores were obtained for students in Spring 2023. FAST was first administered
during the 2022–2023 school year, and only Spring FAST scores were provided, so scaled
scores from the previous version of the Florida English Language Arts standardized test
– Florida Standards Assessment (FSA) – from Spring 2022 were used to establish baseline
equivalence. Spring 2022 FSA was administered in April 2022, and Spring 2023 FAST were
administered between April and May 2023.
Student Demographics. Information on individual student demographic characteristics was also
collected along with each assessment. In particular, grade, gender, ethnicity, race, and disability
status were collected with the WIDA ACCESS assessment, and grade, gender, ethnicity, and race
were collected with the FAST assessment. Note that students could select multiple races.
Imagine Language & Literacy Usage. Program usage data was obtained to determine students’
engagement and progress in Imagine Language & Literacy. These data included the total
minutes students spent in the program and the number of lessons students passed.
2Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
the difference in literacy achievement between treatment and control students. The treatment
group was comprised of all ELL students who logged any usage in the Imagine Language &
Literacy program during the 2022–2023 school year, while the control group included all ELL
students who did not. This study is a quasi-experimental design as assignment to the treatment
and control groups was not random. Statistical procedures were used to ensure baseline
equivalence of the treatment and control samples.
MEASURES
Multiple data sources were compiled to describe students, their performance, and their work
in Imagine Language & Literacy. Student literacy proficiency outcomes were determined using
two standardized assessments. Student demographic data were collected to provide additional
information on student characteristics that may impact measures of learning outcomes. Data from
the Imagine Language & Literacy program were incorporated to evaluate student engagement
in Imagine Language & Literacy. These data sources are reviewed in more detail below.
Literacy Proficiency. Students’ English literacy proficiency was determined using the WIDA
ACCESS 2.0 (WIDA ACCESS) assessment for students in Grades 1 through 5 and the Florida
Assessment of Student Thinking (FAST) for students in Grades 4 and 5.
● WIDA ACCESS overall scale scores were obtained for students in 2022 and 2023. In Florida,
WIDA ACCESS is administered between January and March of each school year. Scores from
2022 were used to establish baseline equivalence between study groups, and 2023 scores
were used to estimate the effect of Imagine Language & Literacy on literacy proficiency.
● FAST scaled scores were obtained for students in Spring 2023. FAST was first administered
during the 2022–2023 school year, and only Spring FAST scores were provided, so scaled
scores from the previous version of the Florida English Language Arts standardized test
– Florida Standards Assessment (FSA) – from Spring 2022 were used to establish baseline
equivalence. Spring 2022 FSA was administered in April 2022, and Spring 2023 FAST were
administered between April and May 2023.
Student Demographics. Information on individual student demographic characteristics was also
collected along with each assessment. In particular, grade, gender, ethnicity, race, and disability
status were collected with the WIDA ACCESS assessment, and grade, gender, ethnicity, and race
were collected with the FAST assessment. Note that students could select multiple races.
Imagine Language & Literacy Usage. Program usage data was obtained to determine students’
engagement and progress in Imagine Language & Literacy. These data included the total
minutes students spent in the program and the number of lessons students passed.
2Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
WIDA ACCESS ANALYTICAL SAMPLE
Separate analytical samples were created for the WIDA ACCESS and FAST analyses. WIDA
ACCESS scores were collected for students in Grades 1–5. A total of 4,633 treatment students
who used Imagine Language & Literacy and 5,458 control students who did not use Imagine
Language & Literacy were initially identified. To ensure that the baseline characteristics of
treatment and control students used in analyses were comparable, 1:1 nearest neighbor
propensity score matching without replacement was used to create a statistically equivalent
analytical sample.1 In the WIDA ACCESS sample, control students were matched to treatment
students based on their Spring 2022 WIDA ACCESS overall scaled score, gender, race, ethnicity,
and disability status. This matching process was completed on each grade individually before
combining the matched grade-level samples to create the total analytical sample. The resulting
WIDA ACCESS analytical sample included 1,381 users of Imagine Language & Literacy and 1,381
non-users. Table 1 below describes the characteristics of the resulting WIDA ACCESS sample.
1 Propensity score matching was executed using the matchit function in R’s MatchIt package with the caliper set to 0.05
for the WIDA ACCESS sample to achieve sufficient baseline equivalence.
3Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Separate analytical samples were created for the WIDA ACCESS and FAST analyses. WIDA
ACCESS scores were collected for students in Grades 1–5. A total of 4,633 treatment students
who used Imagine Language & Literacy and 5,458 control students who did not use Imagine
Language & Literacy were initially identified. To ensure that the baseline characteristics of
treatment and control students used in analyses were comparable, 1:1 nearest neighbor
propensity score matching without replacement was used to create a statistically equivalent
analytical sample.1 In the WIDA ACCESS sample, control students were matched to treatment
students based on their Spring 2022 WIDA ACCESS overall scaled score, gender, race, ethnicity,
and disability status. This matching process was completed on each grade individually before
combining the matched grade-level samples to create the total analytical sample. The resulting
WIDA ACCESS analytical sample included 1,381 users of Imagine Language & Literacy and 1,381
non-users. Table 1 below describes the characteristics of the resulting WIDA ACCESS sample.
1 Propensity score matching was executed using the matchit function in R’s MatchIt package with the caliper set to 0.05
for the WIDA ACCESS sample to achieve sufficient baseline equivalence.
3Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Table 1. Student Characteristics of the WIDA ACCESS Analytical Sample
Group
Comparison
Students
Imagine
Language &
Literacy Students p-value
Standardized
Mean Difference
(SMD)
n 1,381 1,381
Average (SD) Spring 2022 WIDA
ACCESS Overall Scaled Score
254.55
(56.55)
251.91
(59.40) .233 0.045
Grade >.999 <0.001
Grade 1 356 356
Grade 2 434 434
Grade 3 242 242
Grade 4 183 183
Grade 5 166 166
Gender .909 0.006
Female 669 673
Male 712 708
Ethnicity: Hispanic/Latino .568 0.024
No 270 283
Yes 1,111 1,098
Race: Black/African American .066 0.072
No 1,224 1,191
Yes 157 190
Race: White .046 0.078
No 210 250
Yes 1,171 1,131
Has Disability .900 0.007
No 1,242 1,239
Yes 139 142
4Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Group
Comparison
Students
Imagine
Language &
Literacy Students p-value
Standardized
Mean Difference
(SMD)
n 1,381 1,381
Average (SD) Spring 2022 WIDA
ACCESS Overall Scaled Score
254.55
(56.55)
251.91
(59.40) .233 0.045
Grade >.999 <0.001
Grade 1 356 356
Grade 2 434 434
Grade 3 242 242
Grade 4 183 183
Grade 5 166 166
Gender .909 0.006
Female 669 673
Male 712 708
Ethnicity: Hispanic/Latino .568 0.024
No 270 283
Yes 1,111 1,098
Race: Black/African American .066 0.072
No 1,224 1,191
Yes 157 190
Race: White .046 0.078
No 210 250
Yes 1,171 1,131
Has Disability .900 0.007
No 1,242 1,239
Yes 139 142
4Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
FAST ANALYTICAL SAMPLE
FSA and FAST scores were collected for students in Grades 4 and 5. In total, 795 treatment students
who used Imagine Language & Literacy and 2,699 control students who did not use Imagine
Language & Literacy were identified. Similar to the WIDA ACCESS sample, 1:1 nearest neighbor
propensity score matching without replacement was used to create a statistically equivalent
analytical sample.2 In the FAST sample, control students were matched to treatment students
based on their Spring 2022 FSA scaled score, gender, race, and ethnicity. This matching process
was completed on each grade individually before combining the matched grade-level samples
to create the total analytical sample. The resulting FAST analytical sample included 757 users of
Imagine Language & Literacy and 757 non-users. Table 2 below describes the characteristics of
the resulting FAST sample.
2 Propensity score matching was executed using the matchit function in R’s MatchIt package with the caliper set to 0.10 for
the FAST sample to achieve sufficient baseline equivalence.
5Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
FSA and FAST scores were collected for students in Grades 4 and 5. In total, 795 treatment students
who used Imagine Language & Literacy and 2,699 control students who did not use Imagine
Language & Literacy were identified. Similar to the WIDA ACCESS sample, 1:1 nearest neighbor
propensity score matching without replacement was used to create a statistically equivalent
analytical sample.2 In the FAST sample, control students were matched to treatment students
based on their Spring 2022 FSA scaled score, gender, race, and ethnicity. This matching process
was completed on each grade individually before combining the matched grade-level samples
to create the total analytical sample. The resulting FAST analytical sample included 757 users of
Imagine Language & Literacy and 757 non-users. Table 2 below describes the characteristics of
the resulting FAST sample.
2 Propensity score matching was executed using the matchit function in R’s MatchIt package with the caliper set to 0.10 for
the FAST sample to achieve sufficient baseline equivalence.
5Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Table 2. Student Characteristics of the FAST Analytical Sample
Group
Comparison
Students
Imagine
Language &
Literacy Students p-value
Standardized
Mean Difference
(SMD)
n 757 757
Average (SD) Spring 2022 FSA
Scaled Score 269.74 (19.78) 270.17 (20.31) .681 0.021
Grade Level >.999 <0.001
Grade 4 383 383
Grade 5 374 374
Gender .959 0.005
Female 348 350
Male 409 407
Ethnicity: Hispanic/Latino .559 0.033
No 142 152
Yes 615 605
Race: Black .703 0.024
No 655 661
Yes 102 96
Race: White .944 0.007
No 118 120
Yes 639 637
Race: Asian .567 0.039
No 745 741
Yes 12 16
Race: Pacific Islander .682 0.042
No 755 753
Yes 2 4
ANALYTICAL APPROACH
Multiple linear regression was used to evaluate the differences in achievement between Imagine
Language & Literacy users and non-users, controlling for baseline achievement and demographic
covariates. An indicator of whether a student was a control or treatment student was included
in the regression as the primary predictor variable. Using multiple linear regressions after
propensity score matching ensured that any remaining differences in the underlying treatment
and control samples were controlled for by the regression model, effectively isolating the impact
of Imagine Language & Literacy.
6Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Group
Comparison
Students
Imagine
Language &
Literacy Students p-value
Standardized
Mean Difference
(SMD)
n 757 757
Average (SD) Spring 2022 FSA
Scaled Score 269.74 (19.78) 270.17 (20.31) .681 0.021
Grade Level >.999 <0.001
Grade 4 383 383
Grade 5 374 374
Gender .959 0.005
Female 348 350
Male 409 407
Ethnicity: Hispanic/Latino .559 0.033
No 142 152
Yes 615 605
Race: Black .703 0.024
No 655 661
Yes 102 96
Race: White .944 0.007
No 118 120
Yes 639 637
Race: Asian .567 0.039
No 745 741
Yes 12 16
Race: Pacific Islander .682 0.042
No 755 753
Yes 2 4
ANALYTICAL APPROACH
Multiple linear regression was used to evaluate the differences in achievement between Imagine
Language & Literacy users and non-users, controlling for baseline achievement and demographic
covariates. An indicator of whether a student was a control or treatment student was included
in the regression as the primary predictor variable. Using multiple linear regressions after
propensity score matching ensured that any remaining differences in the underlying treatment
and control samples were controlled for by the regression model, effectively isolating the impact
of Imagine Language & Literacy.
6Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
3 There are three types of lessons in Imagine Language & Literacy: literacy, language, and grammar. The number of literacy
lessons are presented here as they tend to be the most highly correlated with growth on English proficiency assessments.
WIDA ACCESS Results
IMAGINE LANGUAGE & LITERACY USAGE
Treatment students spent an average of 11.43 hours in Imagine Language & Literacy and passed
an average of 5.00 literacy lessons.3 Average time in Imagine Language & Literacy varied by
grade level, with the highest average usage in Grade 1 and the lowest average usage in Grade 4.
See Figures 1 and 2 for a distribution of hours and literacy lessons passed by grade.
Figure 1. Distribution of Hours Spent in Imagine Language & Literacy by Grade
50
40
30
10
53
20
2 41
0Hours in Imagine Language & Literacy
Grade
Note: Outliers that fall above 1.5 times the interquartile range are not included in this figure to ensure
readability. The global maximum hours spent in Imagine Language & Literacy is 80.56 hours.
Figure 2. Distribution of Literacy Lessons Passed in Imagine Language & Literacy by Grade
25
20
15
5
53
10
2 41
0
Literacy Lessons Passed
Grade
Note: Outliers that fall above 1.5 times the interquartile range are not included in this figure to ensure
readability. The global maximum number of lessons passed in Imagine Language & Literacy is 45 lessons.
7Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
lessons are presented here as they tend to be the most highly correlated with growth on English proficiency assessments.
WIDA ACCESS Results
IMAGINE LANGUAGE & LITERACY USAGE
Treatment students spent an average of 11.43 hours in Imagine Language & Literacy and passed
an average of 5.00 literacy lessons.3 Average time in Imagine Language & Literacy varied by
grade level, with the highest average usage in Grade 1 and the lowest average usage in Grade 4.
See Figures 1 and 2 for a distribution of hours and literacy lessons passed by grade.
Figure 1. Distribution of Hours Spent in Imagine Language & Literacy by Grade
50
40
30
10
53
20
2 41
0Hours in Imagine Language & Literacy
Grade
Note: Outliers that fall above 1.5 times the interquartile range are not included in this figure to ensure
readability. The global maximum hours spent in Imagine Language & Literacy is 80.56 hours.
Figure 2. Distribution of Literacy Lessons Passed in Imagine Language & Literacy by Grade
25
20
15
5
53
10
2 41
0
Literacy Lessons Passed
Grade
Note: Outliers that fall above 1.5 times the interquartile range are not included in this figure to ensure
readability. The global maximum number of lessons passed in Imagine Language & Literacy is 45 lessons.
7Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Table 3. Overall Impact of Imagine Language & Literacy on 2023 WIDA ACCESS Overall Scaled Score
Variables Estimate Standard Error p-value
Intercept 166.07 2.99 <.001
Imagine Language & Literacy
User Indicator 5.71 0.86 <.001
Grade-Level Indicator
Grade 2 -15.00 1.49 <.001
Grade 3 -6.09 1.64 <.001
Grade 4 6.08 1.91 .001
Grade 5 3.61 1.97 .067
2022 WIDA ACCESS Overall Scaled Score 0.56 0.01 <.001
Male Indicator 0.29 0.86 .738
Ethnicity: Hispanic/Latino Indicator -2.08 1.38 .131
Race: Black/African American Indicator -6.11 2.03 .003
Race: White Indicator -2.14 2.03 .293
Disability Indicator -14.36 1.47 <.001
PROGRAM IMPACT ON STUDENT ACHIEVEMENT
Overall, use of Imagine Language & Literacy was found to generate a positive and statistically
significant impact on students’ English language proficiency. Specifically, students who used
Imagine Language & Literacy scored an average of 5.71 points higher on the 2023 WIDA ACCESS
assessment than otherwise similar non-user students, B = 5.71, t(2750) = 6.663, p < .001. Program
usage and the other covariates in the model accounted for 70% of the variance found in 2023
scores, R2 = .700, F(11,2750) = 582.7, p < .001. The Hedges’ g effect size of Imagine Language &
Literacy program usage is .140.4 Table 3 summarizes the results of the multiple linear regression.
The covariate-adjusted mean 2023 WIDA ACCESS overall scaled score was 299.25 for Imagine
Language & Literacy users and 293.54 for non-users.
4 The effect size is calculated using Hedges’ g computation following What Works Clearinghouse’s Procedures and Standards
Handbook, Version 5.0. The unadjusted standard deviations of the 2023 WIDA ACCESS scores can be found in Appendix A.
8Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Variables Estimate Standard Error p-value
Intercept 166.07 2.99 <.001
Imagine Language & Literacy
User Indicator 5.71 0.86 <.001
Grade-Level Indicator
Grade 2 -15.00 1.49 <.001
Grade 3 -6.09 1.64 <.001
Grade 4 6.08 1.91 .001
Grade 5 3.61 1.97 .067
2022 WIDA ACCESS Overall Scaled Score 0.56 0.01 <.001
Male Indicator 0.29 0.86 .738
Ethnicity: Hispanic/Latino Indicator -2.08 1.38 .131
Race: Black/African American Indicator -6.11 2.03 .003
Race: White Indicator -2.14 2.03 .293
Disability Indicator -14.36 1.47 <.001
PROGRAM IMPACT ON STUDENT ACHIEVEMENT
Overall, use of Imagine Language & Literacy was found to generate a positive and statistically
significant impact on students’ English language proficiency. Specifically, students who used
Imagine Language & Literacy scored an average of 5.71 points higher on the 2023 WIDA ACCESS
assessment than otherwise similar non-user students, B = 5.71, t(2750) = 6.663, p < .001. Program
usage and the other covariates in the model accounted for 70% of the variance found in 2023
scores, R2 = .700, F(11,2750) = 582.7, p < .001. The Hedges’ g effect size of Imagine Language &
Literacy program usage is .140.4 Table 3 summarizes the results of the multiple linear regression.
The covariate-adjusted mean 2023 WIDA ACCESS overall scaled score was 299.25 for Imagine
Language & Literacy users and 293.54 for non-users.
4 The effect size is calculated using Hedges’ g computation following What Works Clearinghouse’s Procedures and Standards
Handbook, Version 5.0. The unadjusted standard deviations of the 2023 WIDA ACCESS scores can be found in Appendix A.
8Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
DIFFERENTIAL IMPACT BY GRADE
A series of analyses were further conducted to examine whether the effects of Imagine Language
& Literacy varied across grade levels. Descriptive tables of unadjusted average WIDA ACCESS
overall scaled scores by grade can be found in Appendix A, and tables demonstrating baseline
equivalence by grade can be found in Appendix B. Imagine Language & Literacy users achieved
statistically significantly higher 2023 WIDA ACCESS overall scaled scores than comparable
non-users for students in Grades 3–5; the difference is non-significant for students in Grades 1
and 2 (Table 4). Complete regression results can be found in Appendix C.
Table 4. Impact of Imagine Language & Literacy on 2023 WIDA ACCESS Overall Scaled
Scores by Grade
Grade
Estimate on Imagine
Language & Literacy
Indicator Variable Standard Error p-value
Grade 1 -0.68 1.58 .669
Grade 2 2.28 1.47 .120
Grade 3 12.35 1.81 <.001
Grade 4 9.41 2.09 <.001
Grade 5 10.58 2.47 <.001
9Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
A series of analyses were further conducted to examine whether the effects of Imagine Language
& Literacy varied across grade levels. Descriptive tables of unadjusted average WIDA ACCESS
overall scaled scores by grade can be found in Appendix A, and tables demonstrating baseline
equivalence by grade can be found in Appendix B. Imagine Language & Literacy users achieved
statistically significantly higher 2023 WIDA ACCESS overall scaled scores than comparable
non-users for students in Grades 3–5; the difference is non-significant for students in Grades 1
and 2 (Table 4). Complete regression results can be found in Appendix C.
Table 4. Impact of Imagine Language & Literacy on 2023 WIDA ACCESS Overall Scaled
Scores by Grade
Grade
Estimate on Imagine
Language & Literacy
Indicator Variable Standard Error p-value
Grade 1 -0.68 1.58 .669
Grade 2 2.28 1.47 .120
Grade 3 12.35 1.81 <.001
Grade 4 9.41 2.09 <.001
Grade 5 10.58 2.47 <.001
9Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
FAST Results
IMAGINE LANGUAGE & LITERACY USAGE
Treatment students spent an average of 11.77 hours in Imagine Language & Literacy and
passed an average of 5.13 literacy lessons. Average time in Imagine Language & Literacy
varied by grade level, with the higher average usage in Grade 4. See Figures 3 and 4 for a
distribution of hours and lessons passed by grade.
Figure 3. Distribution of Hours Spent in Imagine
Language & Literacy by Grade
50
40
30
10
5
20
40Hours in Imagine Language & Literacy
Grade
Figure 4. Distribution of Literacy Lessons Passed
in Imagine Language & Literacy by Grade
25
20
15
5
5
10
40
Literacy Lessons Passed
Grade
Note: Outliers that fall above 1.5 times the interquartile range are not included in this figure to ensure readability.
For Figure 3, the global maximum hours spent in Imagine Language & Literacy is 66.27 hours. For Figure 4, the global
maximum number of lessons passed in Imagine Language & Literacy is 33 literacy lessons.
10Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
IMAGINE LANGUAGE & LITERACY USAGE
Treatment students spent an average of 11.77 hours in Imagine Language & Literacy and
passed an average of 5.13 literacy lessons. Average time in Imagine Language & Literacy
varied by grade level, with the higher average usage in Grade 4. See Figures 3 and 4 for a
distribution of hours and lessons passed by grade.
Figure 3. Distribution of Hours Spent in Imagine
Language & Literacy by Grade
50
40
30
10
5
20
40Hours in Imagine Language & Literacy
Grade
Figure 4. Distribution of Literacy Lessons Passed
in Imagine Language & Literacy by Grade
25
20
15
5
5
10
40
Literacy Lessons Passed
Grade
Note: Outliers that fall above 1.5 times the interquartile range are not included in this figure to ensure readability.
For Figure 3, the global maximum hours spent in Imagine Language & Literacy is 66.27 hours. For Figure 4, the global
maximum number of lessons passed in Imagine Language & Literacy is 33 literacy lessons.
10Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Table 5. Overall Impact of Imagine Language & Literacy on Spring 2023 FAST Scaled Score
Variables Estimate Standard Error p-value
Intercept 93.52 8.21 <.001
Imagine Language & Literacy
User Indicator 2.89 0.96 .003
Grade-Level Indicator
Grade 5 2.72 0.96 .005
Spring 2022 FSA Scaled Score 0.71 0.02 <.001
Male Indicator 0.07 0.97 .944
Ethnicity: Hispanic Indicator -2.21 1.61 .170
Race: Black Indicator 2.48 4.68 .595
Race: White Indicator 2.48 4.50 .581
Race: Asian Indicator 5.12 5.30 .334
Race: Pacific Islander Indicator 6.98 8.52 .413
PROGRAM IMPACT ON STUDENT ACHIEVEMENT
Overall, use of Imagine Language & Literacy was found to generate a positive and statistically
significant impact on students’ FAST performance. Specifically, students who used Imagine
Language & Literacy scored an average of 2.89 points higher on the Spring 2023 FAST assessment
than otherwise similar non-user students, B = 2.89, t(1504) = 3.02, p = .003. Program usage
and the other covariates in the model accounted for 38% of the variance found in Spring 2023
scores, R2 = .380, F(9,1504) = 102.3, p < .001. The Hedges’ g effect size of Imagine Language &
Literacy program usage is .123.5 Table 5 summarizes the results of the multiple linear regression.
The covariate-adjusted mean Spring 2023 FAST overall scaled score was 295.75 for Imagine
Language & Literacy users and 292.86 for non-users.
5 The effect size is calculated using Hedges’ g computation following What Works Clearinghouse’s Procedures and
Standards Handbook, Version 5.0. The unadjusted standard deviations of the Spring 2023 FAST scores can be found in
Appendix A.
11Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Variables Estimate Standard Error p-value
Intercept 93.52 8.21 <.001
Imagine Language & Literacy
User Indicator 2.89 0.96 .003
Grade-Level Indicator
Grade 5 2.72 0.96 .005
Spring 2022 FSA Scaled Score 0.71 0.02 <.001
Male Indicator 0.07 0.97 .944
Ethnicity: Hispanic Indicator -2.21 1.61 .170
Race: Black Indicator 2.48 4.68 .595
Race: White Indicator 2.48 4.50 .581
Race: Asian Indicator 5.12 5.30 .334
Race: Pacific Islander Indicator 6.98 8.52 .413
PROGRAM IMPACT ON STUDENT ACHIEVEMENT
Overall, use of Imagine Language & Literacy was found to generate a positive and statistically
significant impact on students’ FAST performance. Specifically, students who used Imagine
Language & Literacy scored an average of 2.89 points higher on the Spring 2023 FAST assessment
than otherwise similar non-user students, B = 2.89, t(1504) = 3.02, p = .003. Program usage
and the other covariates in the model accounted for 38% of the variance found in Spring 2023
scores, R2 = .380, F(9,1504) = 102.3, p < .001. The Hedges’ g effect size of Imagine Language &
Literacy program usage is .123.5 Table 5 summarizes the results of the multiple linear regression.
The covariate-adjusted mean Spring 2023 FAST overall scaled score was 295.75 for Imagine
Language & Literacy users and 292.86 for non-users.
5 The effect size is calculated using Hedges’ g computation following What Works Clearinghouse’s Procedures and
Standards Handbook, Version 5.0. The unadjusted standard deviations of the Spring 2023 FAST scores can be found in
Appendix A.
11Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
DIFFERENTIAL IMPACT BY GRADE
Analyses were further conducted to examine whether the effects of Imagine Language &
Literacy varied across individual grades. Descriptive tables of unadjusted average FSA and
FAST scaled scores by grade can be found in Appendix A, and tables demonstrating baseline
equivalence by grade can be found in Appendix D. Imagine Language & Literacy users
achieved statistically significantly higher Spring 2023 FAST scaled scores than comparable
non-users for students in Grade 5; the observed difference was non-significant for students in
Grade 4 (Table 6). Complete regression results can be found in Appendix E.
Table 6. Impact of Imagine Language & Literacy on Spring 2023 FAST Overall Scaled
Scores by Grade
Grade
Estimate on Imagine
Language & Literacy
Indicator Variable Standard Error p-value
Grade 4 1.54 1.41 .275
Grade 5 4.22 1.28 .001
12Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Analyses were further conducted to examine whether the effects of Imagine Language &
Literacy varied across individual grades. Descriptive tables of unadjusted average FSA and
FAST scaled scores by grade can be found in Appendix A, and tables demonstrating baseline
equivalence by grade can be found in Appendix D. Imagine Language & Literacy users
achieved statistically significantly higher Spring 2023 FAST scaled scores than comparable
non-users for students in Grade 5; the observed difference was non-significant for students in
Grade 4 (Table 6). Complete regression results can be found in Appendix E.
Table 6. Impact of Imagine Language & Literacy on Spring 2023 FAST Overall Scaled
Scores by Grade
Grade
Estimate on Imagine
Language & Literacy
Indicator Variable Standard Error p-value
Grade 4 1.54 1.41 .275
Grade 5 4.22 1.28 .001
12Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
ConclusionEducational technology provides a means for improving student achievement through
supplemental instruction that is individualized to meet the needs of each student. Moreover,
digital learning solutions have a demonstrated impact on student literacy outcomes for English
language learners (Rahmati et al., 2021) and are a critical component of the future of education
(Haleem et al., 2022). Imagine Language & Literacy offers such a solution.
This study set out to examine the impact of Imagine Language & Literacy on the development
of English language proficiency of ELL students in Grades 1–5. Overall, findings revealed a
statistically significant impact with both WIDA ACCESS and FAST assessments. Students who
used Imagine Language & Literacy scored 5.71 points higher on the 2023 administration of the
WIDA ACCESS assessment and 3.13 points higher on the Spring 2023 administration of the
FAST assessment than did similar comparison students (p < .01).
A limitation of this study includes the lack of baseline scores for students in Grades 1–3 on the
FAST assessment. Particularly, only Grades 4 and 5 were included in FAST analyses since they
are the only grades that also took the FSA assignment in Spring 2022. In the future, use of fall
FAST scores can provide baseline scores to expand the FAST analysis to include additional
grade levels.
In summary, this study provides evidence of effectiveness of Imagine Language & Literacy
on English language proficiency. Specifically, it demonstrates Imagine Language & Literacy’s
impact on the literacy achievement of students in Grades 1–5 on the WIDA ACCESS assessment
and of students in Grades 4–5 on the FAST assessment by comparing the outcomes of students
who participated in the program to those who did not.
13Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
supplemental instruction that is individualized to meet the needs of each student. Moreover,
digital learning solutions have a demonstrated impact on student literacy outcomes for English
language learners (Rahmati et al., 2021) and are a critical component of the future of education
(Haleem et al., 2022). Imagine Language & Literacy offers such a solution.
This study set out to examine the impact of Imagine Language & Literacy on the development
of English language proficiency of ELL students in Grades 1–5. Overall, findings revealed a
statistically significant impact with both WIDA ACCESS and FAST assessments. Students who
used Imagine Language & Literacy scored 5.71 points higher on the 2023 administration of the
WIDA ACCESS assessment and 3.13 points higher on the Spring 2023 administration of the
FAST assessment than did similar comparison students (p < .01).
A limitation of this study includes the lack of baseline scores for students in Grades 1–3 on the
FAST assessment. Particularly, only Grades 4 and 5 were included in FAST analyses since they
are the only grades that also took the FSA assignment in Spring 2022. In the future, use of fall
FAST scores can provide baseline scores to expand the FAST analysis to include additional
grade levels.
In summary, this study provides evidence of effectiveness of Imagine Language & Literacy
on English language proficiency. Specifically, it demonstrates Imagine Language & Literacy’s
impact on the literacy achievement of students in Grades 1–5 on the WIDA ACCESS assessment
and of students in Grades 4–5 on the FAST assessment by comparing the outcomes of students
who participated in the program to those who did not.
13Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
References
Faria, A.-M., Bergey, R., Baird, A. S., & Lishinski, A. (2019). Using technology to support English language learners in
higher education: A study of Voxy’s effect on English language proficiency. American Institutes for Research.
https://www.air.org/resource/using-technology-support-english-language-learners-highereducation-study-voxy-s-effect
Haleem, A., Javaid, M., Qadri, M. A., & Suman, R. (2022). Understanding the role of digital technologies in education:
A review. Sustainable Operations and Computers, 3, 275–285.
National Center for Education Statistics. (2023). English Learners in Public Schools. Condition of Education. U.S.
Department of Education, Institute of Education Sciences. Retrieved January 31, 2024, from https://nces.ed.gov/
programs/coe/indicator/cgf.
Park, C., & Son, J. (2009). Implementing Computer-Assisted Language Learning in the EFL Classroom: Teachers’
Perceptions and Perspectives. https://doi.org/10.5172/ijpl.5.2.80
Pourhossein Gilakjani, A. (2017). A review of the literature of technology into the learning and teaching of English
language skills. International Journal of English Linguistics, 7(5), 95–106. doi: https://doi.org/10.5539/ijel.v7n5p95
Pourhossein Gilakjani, A., & Sabouri, N. B. (2014). Role of Iranian EFL teachers about using Pronunciation Power
software in the instruction of English pronunciation. English Language Teaching, 7(1), 139–148. doi: http://dx.doi.
org/10.5539/elt.v7n1p139
Rahmati, J., Izadpanah, S., & Shahnavaz, A. (2021). A meta-analysis on educational technology in English language
teaching. Language Testing in Asia, 11(7). https://link.springer.com/content/pdf/10.1186/s40468-021-00121-w.pdf
Solanki, D., & Shyamleel, M. P. (2012). Use of technology in English language teaching and learning: An analysis. 2012
International Conference on Language, Medias and Culture IPEDR vol. 33. IACSIT Press. 150–156.
U.S. Department of Education (DOE). (n.d.) Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2022 Reading Assessment.
What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0.
U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and
Regional Assistance (NCEE).
14Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Faria, A.-M., Bergey, R., Baird, A. S., & Lishinski, A. (2019). Using technology to support English language learners in
higher education: A study of Voxy’s effect on English language proficiency. American Institutes for Research.
https://www.air.org/resource/using-technology-support-english-language-learners-highereducation-study-voxy-s-effect
Haleem, A., Javaid, M., Qadri, M. A., & Suman, R. (2022). Understanding the role of digital technologies in education:
A review. Sustainable Operations and Computers, 3, 275–285.
National Center for Education Statistics. (2023). English Learners in Public Schools. Condition of Education. U.S.
Department of Education, Institute of Education Sciences. Retrieved January 31, 2024, from https://nces.ed.gov/
programs/coe/indicator/cgf.
Park, C., & Son, J. (2009). Implementing Computer-Assisted Language Learning in the EFL Classroom: Teachers’
Perceptions and Perspectives. https://doi.org/10.5172/ijpl.5.2.80
Pourhossein Gilakjani, A. (2017). A review of the literature of technology into the learning and teaching of English
language skills. International Journal of English Linguistics, 7(5), 95–106. doi: https://doi.org/10.5539/ijel.v7n5p95
Pourhossein Gilakjani, A., & Sabouri, N. B. (2014). Role of Iranian EFL teachers about using Pronunciation Power
software in the instruction of English pronunciation. English Language Teaching, 7(1), 139–148. doi: http://dx.doi.
org/10.5539/elt.v7n1p139
Rahmati, J., Izadpanah, S., & Shahnavaz, A. (2021). A meta-analysis on educational technology in English language
teaching. Language Testing in Asia, 11(7). https://link.springer.com/content/pdf/10.1186/s40468-021-00121-w.pdf
Solanki, D., & Shyamleel, M. P. (2012). Use of technology in English language teaching and learning: An analysis. 2012
International Conference on Language, Medias and Culture IPEDR vol. 33. IACSIT Press. 150–156.
U.S. Department of Education (DOE). (n.d.) Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2022 Reading Assessment.
What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0.
U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and
Regional Assistance (NCEE).
14Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Appendix A
Table A1. Unadjusted Mean WIDA ACCESS Overall Scaled Score by Grade Band
2022 (SD) 2023 (SD) Mean Change
Grade 1
Imagine Language & Literacy (n = 356) 178.22 (52.84) 263.74 (28.47) 85.53
Comparison (n = 356) 187.68 (54.90) 267.74 (30.85) 80.06
Grade 2
Imagine Language & Literacy (n = 434) 266.53 (30.46) 298.00 (31.39) 31.48
Comparison (n = 434) 266.51 (28.32) 295.77 (33.13) 29.26
Grade 3
Imagine Language & Literacy (n = 242) 270.12 (34.42) 315.67 (32.76) 45.56
Comparison (n = 242) 270.21 (33.47) 303.33 (30.36) 33.12
Grade 4
Imagine Language & Literacy (n = 183) 295.20 (31.68) 340.36 (34.63) 45.16
Comparison (n = 183) 295.41 (31.04) 331.25 (31.78) 35.84
Grade 5
Imagine Language & Literacy (n = 166) 297.51 (36.03) 339.60 (37.42) 42.08
Comparison (n = 166) 298.77 (35.44) 330.48 (33.39) 31.71
Combined Grades 1–5
Imagine Language & Literacy (n = 1,381) 251.91 (59.40) 302.88 (42.55) 50.97
Comparison (n = 1,381) 254.55 (56.55) 298.75 (39.15) 44.20
Table A2. Unadjusted Mean FSA and FAST Score
Spring 2022 FSA Score (SD) Spring 2023 FAST Score (SD)
Grade 4
Imagine Language & Literacy (n = 383) 269.63 (19.94) 286.92 (25.27)
Comparison (n = 383) 269.09 (19.42) 284.86 (21.88)
Grade 5
Imagine Language & Literacy (n = 374) 270.72 (20.69) 291.60 (25.29)
Comparison (n = 374) 270.42 (20.15) 287.17 (20.87)
Combined Grades 4–5
Imagine Language & Literacy (n = 757) 270.17 (20.31) 289.23 (25.37)
Comparison (n = 757) 269.74 (19.78) 286.00 (21.40)
15Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Table A1. Unadjusted Mean WIDA ACCESS Overall Scaled Score by Grade Band
2022 (SD) 2023 (SD) Mean Change
Grade 1
Imagine Language & Literacy (n = 356) 178.22 (52.84) 263.74 (28.47) 85.53
Comparison (n = 356) 187.68 (54.90) 267.74 (30.85) 80.06
Grade 2
Imagine Language & Literacy (n = 434) 266.53 (30.46) 298.00 (31.39) 31.48
Comparison (n = 434) 266.51 (28.32) 295.77 (33.13) 29.26
Grade 3
Imagine Language & Literacy (n = 242) 270.12 (34.42) 315.67 (32.76) 45.56
Comparison (n = 242) 270.21 (33.47) 303.33 (30.36) 33.12
Grade 4
Imagine Language & Literacy (n = 183) 295.20 (31.68) 340.36 (34.63) 45.16
Comparison (n = 183) 295.41 (31.04) 331.25 (31.78) 35.84
Grade 5
Imagine Language & Literacy (n = 166) 297.51 (36.03) 339.60 (37.42) 42.08
Comparison (n = 166) 298.77 (35.44) 330.48 (33.39) 31.71
Combined Grades 1–5
Imagine Language & Literacy (n = 1,381) 251.91 (59.40) 302.88 (42.55) 50.97
Comparison (n = 1,381) 254.55 (56.55) 298.75 (39.15) 44.20
Table A2. Unadjusted Mean FSA and FAST Score
Spring 2022 FSA Score (SD) Spring 2023 FAST Score (SD)
Grade 4
Imagine Language & Literacy (n = 383) 269.63 (19.94) 286.92 (25.27)
Comparison (n = 383) 269.09 (19.42) 284.86 (21.88)
Grade 5
Imagine Language & Literacy (n = 374) 270.72 (20.69) 291.60 (25.29)
Comparison (n = 374) 270.42 (20.15) 287.17 (20.87)
Combined Grades 4–5
Imagine Language & Literacy (n = 757) 270.17 (20.31) 289.23 (25.37)
Comparison (n = 757) 269.74 (19.78) 286.00 (21.40)
15Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Appendix B
Table B1. Grade 1 Baseline Equivalence
Group
Comparison
Students
Imagine
Language &
Literacy Students p-value
Standardized
Mean Difference
(SMD)
n 356 356
Average (SD) 2022 WIDA ACCESS
Overall Scaled Score
187.68
(54.90)
178.22
(52.84) .019 0.176
Gender .154 0.113
Female 163 183
Male 193 173
Ethnicity: Hispanic/Latino .924 0.014
No 69 67
Yes 287 289
Race: Black/African American >.999 0.010
No 328 327
Yes 28 29
Race: White .730 0.034
No 45 41
Yes 311 315
Has Disability .822 0.025
No 309 312
Yes 47 44
16Impact Evaluation of Imagine Language & Literacy in a Florida Public School District
Table B1. Grade 1 Baseline Equivalence
Group
Comparison
Students
Imagine
Language &
Literacy Students p-value
Standardized
Mean Difference
(SMD)
n 356 356
Average (SD) 2022 WIDA ACCESS
Overall Scaled Score
187.68
(54.90)
178.22
(52.84) .019 0.176
Gender .154 0.113
Female 163 183
Male 193 173
Ethnicity: Hispanic/Latino .924 0.014
No 69 67
Yes 287 289
Race: Black/African American >.999 0.010
No 328 327
Yes 28 29
Race: White .730 0.034
No 45 41
Yes 311 315
Has Disability .822 0.025
No 309 312
Yes 47 44
16Impact Evaluation of Imagine Language & Literacy in a Florida Public School District