# Full text of "ERIC ED485398: Saxon Math. What Works Clearinghouse Intervention Report"

## See other formats

What Works Clearinghouse Curriculum-based interventions for increasing K-12 math achievement — middle school Institute of Education Sciences Intervention report Saxon Math updated December i , 2004 Intervention Saxon Math curricula and materials are available for grades K through 12, with the content and skills designed to meet National Council of Teachers of Mathematics (NCTM) and various state standards. Each course lasts for one year, and students participate in 120-lesson courseware packages that last for about 60 minutes a day. Used in teacher-led lessons, Saxon Math is designed to teach in increments, provide continual practice, and test cumulative learning every five lessons. For Middle school students. Findings One randomized controlled trial on Saxon Math found no significant difference in posttest scores between students using Saxon Math and the comparison group using the University of Chicago Mathematics Project NCTM curriculum. One quasi-experimental design study found that students using Saxon Math had higher gains in overall math, math computation, and math concepts compared with other students. However, the study analyzed the data at the wrong level, making it impossible to accurately determine the significance of the finding. Evidence base 1 randomized controlled trial meets evidence standards. y 1 quasi-experimental design study meets evidence standards with reservations. G 4 studies do not meet evidence screens. ( see S y m £> 0 / key on page 7) Evidence limits The evidence base is limited to two studies. The first is a randomized controlled trial of 8th-grade students in a rural-suburban Nebraska junior high school. A second study is a quasi-experimental design study of 8th-grade students in Oklahoma City middle schools. Quasi-experimental studies provide weaker evidence of effects because it is possible that unmeasured differences between the groups affected the findings. Further, this study analyzed the data at the wrong level, which may bias the findings. The samples for both studies were small (36-78 students). Four studies do not meet evidence standards. Scope of use Saxon Math’s first textbook (Algebra I for 9th grade) was implemented in 1980, and Saxon Algebra 1/2 (8th grade) was implemented in 1986. Information is not available about the number and demographics of students, schools, or districts using the intervention. Developer and contact Saxon Publishers, www.saxonpublishers.com ; email: info@saxonDublishers.com : telephone: (800) 284-7019. WWC Intervention Reports Profile Saxon Math focuses on fundamental mathematics skills, targeting children from kindergarten through grade 12. This report focuses on middle school math, defined as grades 6 through 9. The 6th grade curriculum covers simplifying expressions containing parentheses, graphing functions, and understanding ratios and proportions. The 7th grade curriculum covers pre-algebra topics such as rate, powers, roots, and geometric proofs. The 8th grade curriculum covers all topics usually taught in pre-algebra in addition to topics from geometry and discrete mathematics. The 9th grade curriculum covers all topics usually taught in a first-year algebra course (such as exponents, roots, and algebraic word problems) as well as conceptual understanding, procedural fluency, strategic competence, adaptive reasoning, and productive disposition. As stated by the developers, Saxon Math covers all five content and skill areas of NCTM standards and meets various state standards. Teaching Each grade of Saxon Math consists of 1 20 daily lessons and 1 2 activity-based investigations. A daily lesson consists of warm-up (10-15 minutes), introduction to the new concept (5-10 minutes), practice focusing on new concept (5 minutes), and mixed practice focusing on new and previously learned concepts (20-30 minutes). Students are introduced to concepts incrementally, given opportunities for continual review and practice, and assessed cumulatively and frequently (every fifth lesson). An assessment score of 80% or lower indicates a need for remediation, and a provision for remediation is part of the program. Lessons are designed to be one hour daily (this includes practice and review time), and assessments occur every fifth lesson, usually on Friday each week. The teacher is responsible for facilitating and mediating the warm-up session, introducing the new concept, and conducting the practice sessions. Teachers introduce the daily concept using manipulatives or representative models accompanied by the procedures needed to solve the problem. Teachers are instructed to conduct lessons in sequence, not skip lessons, limit direct instruction to 10-15 minutes of group (or individual) instruction, spend the majority of class time allowing children to do mathematics problems in the problem sets, and assign all problems in each set. Supports are available for teachers of Saxon Math. Each state has an educational representative. The curriculum developers have a comprehensive Web site offering general information, resource materials, and an email address for questions. Other supports include customer service representatives, in-service training, telephone teacher support, a helpline, teachers’ resource booklets, in-service videos, and administrator’s guides to help principals and administrators implement Saxon Math in their classrooms. Scope of use Saxon Math’s first textbook (Algebra I for 9th grade) was implemented in 1980 and Saxon Algebra 1/2 (8th grade) was implemented in 1986. Information about the number and demographics of students, schools, or districts currently using the intervention is not available from the Web site or the customer service or educational representatives. Cost The student text costs approximately $50. Additional materials, including the teacher’s manual, can total approximately $380-$900 more. WWC Intervention Reports 2 Study findings Strength of the evidence base References WWC Intervention Reports Randomized controlled trial The single randomized controlled trial on Saxon Math (Peters 1992) found no significant difference in posttest scores between students in Saxon Math and the comparison group when controlling on pretest. The intervention group scored slightly but not significantly higher than the comparison group. There was no evidence that the Saxon Math intervention was more or less effective than the comparison curriculum, the University of Chicago Mathematics Project NCTM curriculum. Quasi-experimental design The single quasi-experimental design study on Saxon Math (Crawford & Raia 1986) found that students in the intervention group made significant gains in overall math and math computation scores but not on math concepts scores, compared with the comparison group. Because of the limitations in the way the analysis was conducted, it is not possible to determine whether these findings are due to the curriculum or to chance. The WWC collected more than 800 studies for the Middle School Math Curriculum review. Six looked at the effects of Saxon Math. One study, a small randomized controlled trial without serious problems, met WWC evidence standards. A second study, a small quasi-experimental design study without serious problems, met WWC evidence standards with reservations. The remaining four studies did not meet WWC evidence screens. In three of these studies, there was only one intervention and one compari- son unit, so the analysis could not separate the effects of the intervention from other factors. The fourth study, a quasi- experimental design study, does not account for pre-existing differences between groups with matching or equating. Studies were rated according to the strength of their causal evidence. Studies that placed students into the intervention and comparison groups randomly (randomized controlled trials) without notable design or implementation flaws are classified as meeting evidence standards (@ a ). Other studies that use comparison groups (quasi-experimental designs) and randomized control trials with notable flaws are classified as meeting evidence standards with reservations ( ✓ a ). Studies are further rated for intervention fidelity, outcome measures, breadth of evidence, reporting on subgroups, analysis, and statistical reporting. That information is provided in study reports, but does not affect the overall rating. In both studies, the interventions were well designed and implemented — and both studies used nationally normed, standardized tests. Neither study looked at all important groups of students or settings. There were several issues with the analysis. The studies were small — the randomized controlled trial had 36 students and the quasi-experimental design study had 78 students. Further, some students in the RCT switched groups, and findings from the QED study should be viewed with caution because of problems with the analysis. Tables A3-A4 describe the outcome studies conducted on Saxon Math that meet WWC evidence standards and meet WWC evidence standards with reservations. For a more detailed description of the study, see the Detailed Study Reports or Brief Study Reports . a See symbol key on page 7. © Peters, K. G. (1992). Skill performance comparability of two algebra programs on an eighth-grade population. Unpublished doctoral dissertation, University of Nebraska, Lincoln. ✓ Crawford, J., & Raia, F. (1986, February). Analyses of eighth grade math texts and achievement (evaluation report). Oklahoma City: Planning, Research, and Evaluation Department, Oklahoma City Public Schools. Q Clay, D.W. (1 998). A study to determine the effects of a non-traditional approach to algebra instruction on student achievement. Master’s thesis, Salem-Teikyo University. (ERIC Document Reproduction Service No. ED428963) © Lafferty, J.F. (1996). The links among mathematics text, students’ achievement, and students’ mathematics anxiety: A comparison of the incremental development and traditional References (continued) texts. Dissertation Abstracts International, 56 (08), 301 4A. (UMI No. 9537085) O Rentschler, R.V. (1994). The effects of Saxon’s incremental review on computational skills and problem-solving achievement of sixth-grade students. Dissertation Abstracts International, 56 (02), 484A. (UMI No. 9518017) G Saxon, J. (1982). Incremental development: A breakthrough in mathematics. Phi Delta Kappan, 63 (4), 482-484. Table 1 The Saxon Math group seemed to score better than the difference between the two groups was somewhere between Effects comparison group, but the difference is not significant. For the -0.54 (favoring the comparison group) and 0.76 (favoring the randomized controlled trial, we are 95% confident that the Saxon Math group). Study Measure Favors Comparison Favors comparison group group mean Saxon Math group -2.00 -1.50 -1.00 -0.50 0 0.50 1.00 1.50 2.00 © Peters Orleans-Hanna Algebra 1992 Prognosis Test 3 (N=36 students) 0.11 / Crawford CAT Overall Math b & Raia (N=78 students 0 ) 1986 0 . 41 d CAT Math Concepts 13 (N=78 students 0 ) 0 . 28 d CAT Math Computation 13 (N=78 students 0 ) 0 . 44 d Approximate percentile ranking 2 % 7° h 16% 31% 50% 69% 84% 93% 98% a A nationally normed, standardized test, b California Achievement Test, a nationally normed, standardized test, c Sample size reported is unit of analysis, not unit of assignment. d When there is no solid line, the study did not provide data to correctly compute the confidence interval. How to read this table: The wide, shaded bar indicates both the direction and estimated size of the effect of the intervention. The estimated effects reported here are standardized differences in the mean values between the intervention and comparison groups. Bars extending to the right of zero denote estimated effects that favor the intervention group and those extending to the left of zero denote estimated effects that favor the comparison group. The solid line through the shaded bar marks the 95% confidence interval of the estimated effect. When the line does not cross zero (and the bar is solid, not striped), the estimate is statistically significant. The bar is striped if the effect is not significant or if significance could not be accurately computed. The scale at the bottom of the chart indicates the approximate percentile distribution of students in the control group. The percentile ranking at the end of the shaded bar can be used to interpret the standardized mean difference in the outcome. For example, an effect of .5 is roughly equivalent to an increase in the mean value from that of the average student in the comparison group (50th percentile) to that of the average student at the 69th percentile. What Works Clearinghouse The What Works Clearinghouse (www.whatworks.ed.gov) was established in 2002 by the U.S. Department of Education’s Institute of Education Sciences to provide educators, policymakers, researchers, and the public with a central and trusted source of scientific evidence of what works in education. Please email all questions and comments to info@whatworks.ed.gov. The What Works Clearinghouse is administered by the U.S. Department of Education through a contract to a joint venture of the American Institutes for Research and the Campbell Collaboration. Appendix Table ai Summary characteristics and findings from randomized controlled trials on Saxon Math Study Study sample Measure Sample size Mean outcome Standard deviation 3 Estimated impact b Intervention Comparison group group Total Intervention group Comparison group Intervention group Comparison group Mean difference Standardized mean difference Peters 8th grade [General] 19 17 36 95.6 95.1 4.53 4.09 0.5 0.11 (±0.65) 1992 “math-talented” achievement students students test c Table A2 Summary characteristics and findings from quasi-experimental design studies on Saxon Math Sample size Mean outcome Standard deviation 3 Estimated impact b Study Study sample Measure Intervention Comparison Intervention group group Total group Comparison group Intervention group Comparison group Mean difference Standardized mean difference Crawford & Raia 8th grade students CAT d Overall Math 39 39 78 55.56 students 6 50.72 11.86 11.75 4.84 0.41 f 1986 8th grade students CAT d Math Concepts 39 39 78 53.36 students 6 49.82 12.44 12.40 3.54 0.20 f 8th grade students CAT d Math Computation 39 39 78 57.59 students 6 51.51 13.35 14.14 6.08 0.44 f a Shows how dispersed the participants’ outcomes are. A small standard deviation would suggest that participants had similar outcomes, b The WWC computed standardized effects, using statistics reported by the study author, c A 60-item nationally normed, standardized test to predict student success in future algebra study, d California Achievement Test. e The sample size reported is unit of analysis, not unit of assignment. f The unit of analysis did not match the unit of assignment, so accurate confidence intervals could not be computed. WWC Intervention Reports 5 Table A3 Characteristics of interventions in reviewed studies on Saxon Math: Peters 1992 Evidence base rating 3 Characteristic Description 0 Study citation Peters, K. G. (1 992). Skill performance comparability of two algebra programs on an eighth-grade population. Unpublished doctoral dissertation, University of Nebraska, Lincoln, NE. Participants 36 8th-grade students. All the students were “math-talented” based on teacher recommendations, prior academic achievement, and personal maturity. Setting Junior high school in a rural suburban district abutting Lincoln, Nebraska; students randomly assigned to one of two classrooms (one intervention classroom and one comparison classroom). The same teacher taught both the intervention and comparison groups. Intervention Participants in the intervention group were taught using the Saxon Math curriculum for 8th grade students (Algebra 1/2). Students in this group participated in 60-minute daily sessions for one year. In each session, the teacher introduced a new concept incrementally, and students had opportunities to practice the new concept and past concepts during each session. Students were assessed every fifth lesson. The intervention is designed to cover 1 20 lessons in one year. Comparison Participants in the comparison group were taught using an NCTM standards based curriculum called the University of Chicago Mathematics Project designed to: build independent learners and thinkers, build understanding of math vocabulary (such as mathematical signs), emphasize reviewing concepts within existing lessons, and increase student comprehension. Primary outcomes and measurement The primary outcome measure is the Orleans-Hanna Algebra Prognosis Test, a nationally normed, valid, and reliable 60-item test designed to predict student success in future algebra study. Teacher training Teacher training was not reported for this study, but teacher resources are available at the Saxon website, including telephone and email access to customer service and educational representatives (in each state). a See symbol key on page 7. WWC Intervention Reports 6 Table A4 Characteristics of interventions in reviewed studies on Saxon Math: Crawford & Raia 1986 Evidence base rating Characteristic Description • Study citation Crawford, J., & Raia, F. (1986, February). Analyses of eighth grade math texts and achievement (evaluation report). Oklahoma City: Planning, Research, and Evaluation Department, Oklahoma City Public Schools. Participants 78 8th grade students matched on pretest California Achievement Test (CAT) scores. Setting Four middle schools in the Oklahoma City Public Schools; four teachers taught both the intervention and the comparison groups. Intervention Participants in the intervention group were taught using the Saxon Math curriculum for 8th-grade students (Algebra 1/2). Specific information about the level of implementation was not provided. The intervention is designed to cover 120 lessons across a one-year period with students participating in daily lessons, approximately 60 minutes a lesson. Students participated during the 1984/85 academic year. Comparison Participants in the comparison group were taught using the Scott-Foresman Mathematics curriculum. Information about this curriculum, including implementation, was not provided. Primary outcomes and measurement The primary outcome measure is the California Achievement Test (CAT), including overall scores and scores for math concepts and math computation. The CAT is a nationally normed, valid, and reliable test designed to measure achievement in the basic skills taught in school. Teacher training Teacher training was not reported for this study, but teacher resources are available at the Saxon website, including telephone and email access to customer service and educational representatives (in each state). Symbol key for evidence base rating 0 Study meets evidence standards (randomized controlled trial without notable flaws). V Study meets evidence standards with reservations (randomized controlled trial with notable flaws or quasi-experimental design study without notable flaws). © Study does not meet evidence screens. WWC Intervention Reports