Comprehensive School Reform Models

AIM at Middle-Grades Results • Different Ways of Knowing
Making Middle Grades Work • Making Schools Work • Middle Start
Success for All Middle School Program • Turning Points

Research Findings

Study 1
Middle School Mathematics: Individual Growth on the Stanford 9. We have recently completed analyses of the impact of the Talent Development Middle School Math Program on students' achievement growth during the first four years of the program in the nation's first three TDMS schools (Balfanz, Byrnes, & Mac Iver, 2002). These analyses cover achievement growth data for 14,258 students from 1997-1998 through 2000-2001 in our three original schools in Philadelphia and their matched comparison sites. First, we checked to make sure that there were no pre-existing differences between the treatment and control groups before these groups entered middle school. Both groups had a mean math problem solving normal curve equivalent score of 36 as 3rd graders and of 37 as 4th graders. Thus, the assumption that the groups had equal prior achievement levels and equal prior achievement growth rates seems valid. Second, we modeled the growth curve of each student using a 3-level HLM model. At level 1 (within-student growth model), math achievement is modeled as a function of an intercept (baseline achievement at third grade), summer drop off, growth during 4th grade, growth during fifth grade, growth during sixth grade, growth during seventh grade, and growth during 8th grade. At level 2 (between students), we modeled the impact of special education status on students' growth in 5th, 6th, 7th, and 8th grade. Finally, at level 3 (between schools) we modeled the impact of attending a Talent Development Middle School on students' growth rates in 5th, 6th, 7th, and 8th grades. Any student with at least one test record at one of the three original TDMS schools or comparison schools was included in the model.

The results revealed that students in TDMS schools showed significantly greater growth in mathematics problem solving achievement on the Stanford 9 than did students in the comparison schools. The TDMS advantage in growth rate was about 7 NCEs in sixth- and eighth-grade (an effect size around one-third of a standard deviation) and was over 10 NCEs in seventh-grade (an effect size of almost one-half of a standard deviation.) On the other hand, TDMS schools and comparison schools were not significantly different in student growth in math procedures achievement on the Stanford 9.

Study 2
Middle School Mathematics: Individual Growth on the Pennsylvania System of Student Assessment. This study compared the achievement growth in mathematics across 3 years at 3 TDMS schools and at 3 matched comparison schools from 1997 to 2000. The594 students in the treatment group and 437 students in the control groups were equivalent at the start (mean normal curve equivalent was 27 for the treatment group and 26 for the control group, t (1029) = 1.30, p = .20.) The treatment group gained an average of 4.0 NCES during these 3 years but the control group gained only 1.9 NCEs, t (1029) = 2.81, p = .005, effect size =.17

Study 3
Middle School Science: Individual Growth on the Stanford 9. An initial evaluation of the TDMS Science Program has been completed using a quasi-experimental design involving 3 pairs of matched middle schools (matched on student demographics, past student achievement, and socioeconomic status.) These analyses involve 2214 students from two different cohorts. The first cohort experienced the pilot version of the TDMS science program.. The second cohort experienced the full version of the TDMS science program. Analyses combining both cohorts reveal that the number of years a student had been exposed to the TDMS science program (0 years in the control schools, up to 2.75 years in TDMS schools) was a significant predictor of students' growth in Science Achievement between the spring of 4th and the spring of 7th grade (b = .06, p < .05). As expected, analyses focusing just on students in the second cohort indicate that the impact of exposure was even larger in this cohort (b=.16, p < .01). Students in the second cohort in TDMS schools received science instruction from teachers who had gained experience in implementing TDMS science. These teachers had also received more continuous and intensive professional development and in-classroom assistance than had the teachers of students in the first cohort. The impact of the full science program can be summarized as an effect size. Specifically, the effect of the TDMS program on the science achievement growth for seventh-graders who had received exposure to the full TDMS science program since the beginning of fifth grade was .39 standard deviations.

Study 4
Middle School Reading: Individual Growth on the Pennsylvania System of Student Assessment. This study (Mac Iver, Byrnes, & Balfanz, 2002) compared the achievement growth in reading across 3 years at 2 TDMS schools and at 2 matched comparison schools from 1997 to 2000. The 216 students in the treatment group and 124 students in the control groups were equivalent at the start (mean normal curve equivalent was 27 for both groups) The treatment group gained an average of 5.8 NCEs during these 3 years but the control group gained only 1.5 NCEs, t (338) = 2.72, p = .004, effect size =.29.

Study 5
Pilot Test of 8th-Grade U.S. History Program. The preliminary evaluation of the Civil War and post-Civil War portion of the TDMS U.S. History program involved 223 eighth-graders drawn from nine U.S. History sections taught by nine different teachers. In hierarchical linear models controlling for prior achievement, students in TDM schools outscored students in the waiting list control school by 1.2 standard deviations (getting about 3 more items right, p < .05) on the 30-item history/citizenship/geography multiple choice test that was originally designed for and used in the NELS:88 study conducted by the U.S. Department of Education. This indicates that TDMS students are leaving middle school with significantly higher levels of historical literacy than students in the control school.

One of the important thrusts of the TDMS model is to assist schools in increasing the length of real instructional time for the core academic subjects (See Balfanz, Ruby, and Mac Iver, in press). For example, TD's professional development sequences for principals and for teachers emphasize the importance of providing sufficient U.S. history instructional time not just because it increases students' historical literacy but also because history class provides a golden opportunity to build students' reading comprehension of nonfiction materials and their understanding of reading, study, and research strategies appropriate in dealing with factual material. This study revealed that 8th-grade teachers in TDMS schools devoted 77% more time to history and social studies instruction than control classrooms (163 versus 92 hours on average, p =.03, effect size = 1.2 standard deviations) and that the hours of history/social studies instruction provided was an important predictor of student performance on the history/citizenship/geography test (standardized gamma coefficient = .56).

The results from the open-ended subtests suggest that the performance of students in classrooms with heavy implementation of the recommended curriculum were especially likely to outshine those of students in other classrooms. For example, heavy implementation classes outperformed control classes by 1.32 standard deviations on the Civil War subtest. Similarly, heavy implementation classes outperformed control classes by .85 standard deviations on the "Civil War through 1918" subtest.

The results also showed that TDMS classrooms de-emphasized recent history (1920 through the present). Specifically, only one of the six TDMS classrooms covered this period in any substantial manner Perhaps as a consequence, the other five TDMS classrooms under performed control classes on the "1920 through present" subtest by 1.5 standard deviations.

Overall, our preliminary evaluation of the eighth-grade TDMS history program is encouraging. TD classrooms significantly outperformed control classrooms on a broad multiple choice exam, and heavy implementation classes performed better than control classes on two portions of the open-ended test.

<< Background | << Model Design

[top]

View other Comprehensive School Reform Models

AIM at Middle-Grades Results • Different Ways of Knowing
Making Middle Grades Work • Making Schools Work • Middle Start
Success for All Middle School Program • Turning Points