Schooled on Value
Teacher Performance Pay, Coaching, and Long-Run Student Outcomes
Sarah Cohodes, Ozkan Eren & Orgul Ozturk
NBER Working Paper, March 2023
Abstract:
This paper examines the effects of a comprehensive performance pay program for teachers implemented in high-need schools on students’ longer-run educational, criminal justice, and economic self-sufficiency outcomes. Using linked administrative data from a Southern state, we leverage the quasi-randomness of the timing of program adoption across schools to identify causal effects of the school reform. The program improved educational attainment and reduced both criminal activity and dependence on government assistance in early adulthood. We find little scope for student sorting or changes in the composition of teacher workforce, and that program benefits far exceeded its costs. We propose mechanisms for observed long-run effects and provide evidence consistent with these explanations. Several robustness checks and placebo tests support our findings.
An astonishing regularity in student learning rate
Kenneth Koedinger et al.
Proceedings of the National Academy of Sciences, 28 March 2023
Abstract:
Leveraging a scientific infrastructure for exploring how students learn, we have developed cognitive and statistical models of skill acquisition and used them to understand fundamental similarities and differences across learners. Our primary question was why do some students learn faster than others? Or, do they? We model data from student performance on groups of tasks that assess the same skill component and that provide follow-up instruction on student errors. Our models estimate, for both students and skills, initial correctness and learning rate, that is, the increase in correctness after each practice opportunity. We applied our models to 1.3 million observations across 27 datasets of student interactions with online practice systems in the context of elementary to college courses in math, science, and language. Despite the availability of up-front verbal instruction, like lectures and readings, students demonstrate modest initial prepractice performance, at about 65% accuracy. Despite being in the same course, students’ initial performance varies substantially from about 55% correct for those in the lower half to 75% for those in the upper half. In contrast, and much to our surprise, we found students to be astonishingly similar in estimated learning rate, typically increasing by about 0.1 log odds or 2.5% in accuracy per opportunity. These findings pose a challenge for theories of learning to explain the odd combination of large variation in student initial performance and striking regularity in student learning rate.
Post COVID-19 Test Score Recovery: Initial Evidence from State Testing Data
Clare Halloran et al.
NBER Working Paper, April 2023
Abstract:
The COVID-19 pandemic caused significant disruption in schooling in the U.S., and student test scores showed dramatic declines by the end of the 2020-21 school year. We use state test score data to analyze patterns of test score recovery over the 2021-22 school year. On average, we find that 20% of test score losses are recovered in English language arts (ELA) by 2022, compared to 37% in math. These recovery rates do not significantly vary across demographic characteristics, baseline achievement rates, in-person schooling rates in the pandemic school year, or category-based measures of recovery funding allocations. We observe large state-level variation in recovery rates in ELA -- from full recovery to further losses. This evidence suggests state-level factors play an important role in students' academic recovery, but we are unable to isolate particular state factors. Future work should focus on this variation to facilitate a broader recovery effort.
The Effects of Comprehensive Educator Evaluation and Pay Reform on Achievement
Eric Hanushek et al.
NBER Working Paper, March 2023
Abstract:
A fundamental question for education policy is whether outcomes-based accountability including comprehensive educator evaluations and a closer relationship between effectiveness and compensation improves the quality of instruction and raises achievement. We use synthetic control methods to study the comprehensive teacher and principal evaluation and compensation systems introduced in the Dallas Independent School District (Dallas ISD) in 2013 for principals and 2015 for teachers. Under this far-reaching reform, educator evaluations that are used to support teacher growth and determine salary depend on a combination of supervisor evaluations, student achievement, and student or family survey responses. The reform replaced salary scales based on experience and educational attainment with those based on evaluation scores, a radical departure from decades of rigid salary schedules. The synthetic control estimates reveal positive and significant effects of the reforms on math and reading achievement that increase over time. From 2015 through 2019, the average achievement for the synthetic control district fluctuates narrowly between -0.27 s.d. and -0.3 s.d., while the Dallas ISD average increases steadily from -0.28 s.d. in 2015 to -0.08 s.d. in 2019, the final year of the sample. Though the increase for reading is roughly half as large, it is also highly significant.
Findings on Summer Learning Loss Often Fail to Replicate, Even in Recent Data
Joseph Workman, Paul von Hippel & Joseph Merry
Sociological Science, March 2023
Abstract:
It is widely believed that (1) children lose months of reading and math skills over summer vacation and that (2) inequality in skills grows much faster during summer than during school. Concerns have been raised about the replicability of evidence for these claims, but an impression may exist that nonreplicable findings are limited to older studies. After reviewing the 100-year history of nonreplicable results on summer learning, we compared three recent data sources (ECLS- K:2011, NWEA, and Renaissance) that tracked U.S. elementary students’ skills through school years and summers in the 2010s. Most patterns did not generalize beyond a single test. Summer losses looked substantial on some tests but not on others. Score gaps—between schools and students of different income levels, ethnicities, and genders—grew on some tests but not on others. The total variance of scores grew on some tests but not on others. On tests where gaps and variance grew, they did not consistently grow faster during summer than during school. Future research should demonstrate that a summer learning pattern replicates before drawing broad conclusions about learning or inequality.
Classroom Competition, Student Effort, and Peer Effects
Mark Rosenzweig & Bing Xu
NBER Working Paper, April 2023
Abstract:
This paper studies how rewards based on class rank affect student effort and performance using a game-theoretic classroom competition model and data from the resettlement of Southeast Asian refugees in the US. The paper finds that variation in the presence of strong or weak students changes the incentives and test scores of incumbent students depending on their ability group in accord with the competition model, with increases in the number of strong students lowering effort among strong and weak incumbents but raising the test scores of weak incumbents. The results suggest that competition induced by rank-based rewards within homogeneous ability groups lowers overall effort levels, while the presence of strong students directly augments the performance, but not the effort levels, of weak students despite the competition. The paper also rules out a number of alternative explanations for these school composition effects, including disruptions, teacher-initiated changes in curriculum in response to changing class composition, selective incumbent-student school exit, and endogenous responses of refugee location choices to school performance.
Randomized design evidence of the attendance benefits of the EPA School Bus Rebate Program
Meredith Pedde et al.
Nature Sustainability, forthcoming
Abstract:
Approximately 25 million children ride buses to school in the United States. While school buses are the safest school transport from an accident perspective, older buses often expose students to high levels of diesel exhaust. Because these exposures can adversely impact health, which may lead to more missed school, the US Environmental Protection Agency (EPA) has spent millions of dollars to hasten the transition of school bus fleets to cleaner vehicles. Here, we leveraged the randomized allocation of the EPA’s 2012–2017 School Bus Rebate Program funding to causally assess the district attendance impacts of upgrading buses. Districts randomly selected for funding had greater attendance improvements after the lottery than unselected districts, resulting in over 350,000 estimated additional student days of attendance each year (95% confidence interval = −70,678 to 772,865) due to the use of EPA funds. Attendance improvements were greatest when the oldest buses were replaced and for districts with high ridership on applicant buses. Extrapolating our results nationwide, we expect that the replacement of all pre-2000 model year school buses would lead to more than 1.3 million additional student days of attendance per year (95% confidence interval = 247,443 to 2,406,511). Given the importance of attendance to educational success, we conclude that increasing the pace at which older, highly polluting buses are replaced positively impacts student attendance.
Attracting and Retaining Highly Effective Educators in Hard-to-Staff Schools
Andrew Morgan et al.
NBER Working Paper, March 2023
Abstract:
Efforts to attract and retain effective educators in high poverty public schools have had limited success. Dallas ISD addressed this challenge by using information produced by its evaluation and compensation reforms as the basis for effectiveness-adjusted payments that provided large compensating differentials to attract and retain effective teachers in its lowest achievement schools. The Accelerating Campus Excellence (ACE) program offers salary supplements to educators with records of high performance who are willing to work in the most educationally disadvantaged schools. We document that ACE resulted in immediate and sustained increases in student achievement, providing strong evidence that the multi-measure evaluation system identifies effective educators who foster the development of cognitive skills. The improvements at ACE schools were dramatic, bringing average achievement in the previously lowest performing schools close to the district average. When ACE stipends are largely eliminated, a substantial fraction of highly effective teachers leaves, and test scores fall. This highlights the central importance of the performance-based incentives to attract and retain effective educators in previously low-achievement schools.
Natural Resource Booms, Human Capital, and Earnings: Evidence from Linked Education and Employment Records
Alina Kovalenko
American Economic Journal: Applied Economics, April 2023, Pages 184-217
Abstract:
Using administrative panel data on the universe of Texas public school students, I analyze how shocks to local economic conditions affect education and employment decisions. I find that high school students at the bottom of the academic ability distribution worked and earned more in response to the fracking boom and that these earnings gains persisted through ages 24–25 despite the fact that the same students also became less likely to attend classes and graduate from high school. My results suggest that the opportunity cost of education is large for these students.
What Teachers’ Unions Do for Teachers When Collective Bargaining is Prohibited
Eunice Han & Jeffrey Keefe
Labor Studies Journal, forthcoming
Abstract:
Our research examines the relationship between teachers’ unions, teacher compensation, employment conditions, and turnover in Southern US states that prohibit the collective bargaining of public school teachers. We assess union strength by meet-and-confer status and the teacher union density of districts and estimate union impact using propensity score matching. We find that teachers’ unions are positively associated with teacher compensation and employment conditions, even in the absence of collective bargaining. Districts with strong unions have higher dismissals of nontenured teachers for poor performance and a lower attrition rate of qualified teachers, compared to districts with weak unions. This study shows that teachers’ unions in the United States still organize without formal labor-management institutions and make a significant impact on teachers’ work lives, even in a hostile legal environment toward unions.
Are Students Time Constrained? Course Load, GPA, and Failing
Aaron Phipps & Alexander Amaya
U.S. Military Academy Working Paper, March 2023
Abstract:
Given the simultaneous rise in time-to-graduation and college GPA, it may be that students reduce their course load to improve their performance. Yet, evidence to date only shows increased course loads increase GPA. We provide a model showing many unobservable factors -- beyond student ability -- can generate a positive relationship between course load and GPA unless researchers control student schedules. West Point regularly implements the ideal experiment by randomly modifying student schedules with additional training courses. Using 19 years of administrative data, we provide the first causal evidence that taking more courses reduces GPA and increases course failure rates, sometimes substantially.
University Misreporting of Student Metrics: Evidence from the Labor Market for Professional Football Players
Beau Grant Barnes
Washington State University Working Paper, March 2023
Abstract:
The success of universities is increasingly tied to student and program metrics, making the manipulation of such metrics both more prevalent and more consequential. This environment challenges universities’ truth-seeking missions, which are paramount to the value of universities. This paper provides insights into this problem by examining the accuracy of student athlete heights, as reported in university publications. Using a sample of 1,334 student-athletes who participated in the National Football League’s (NFL) Annual Scouting Combine (“the Combine”) between 2018 and 2022, I measure height inflation by comparing the rigorous and independent height measurements taken at the Combine against the heights reported by universities in their media guides. On average, universities report student-athlete heights consistent with customary rounding conventions; however, there is considerable variation in the accuracy of reported student athlete heights, and more than 20 percent of heights are inflated by more than an inch. I find that the accuracy of reported heights is predictable, with universities tending to report inflated heights for student-athletes who are short for their position. Further, I find that universities that regularly send student-athletes to participate in the Combine are less likely to report inflated heights. Evidence also suggests that height inflation is non-trivial to the labor market for professional football players. Specifically, relative to other Combine participants, those with inflated heights are far less likely to be drafted into the NFL. Taken in total, the findings suggest that university misreporting of basic labor-market-relevant facts is widespread and can mislead labor market participants.