Blinded by Justice
Algorithmic Recommendations and Human Discretion
Victoria Angelova, Will Dobbie & Crystal Yang
NBER Working Paper, September 2023
Abstract:
Human decision-makers frequently override the recommendations generated by predictive algorithms, but it is unclear whether these discretionary overrides add valuable private information or reintroduce human biases and mistakes. We develop new quasi-experimental tools to measure the impact of human discretion over an algorithm on the accuracy of decisions, even when the outcome of interest is only selectively observed, in the context of bail decisions. We find that 90% of the judges in our setting underperform the algorithm when they make a discretionary override, with most making override decisions that are no better than random. Yet the remaining 10% of judges outperform the algorithm in terms of both accuracy and fairness when they make a discretionary override. We provide suggestive evidence on the behavior underlying these differences in judge performance, showing that the high-performing judges are more likely to use relevant private information and are less likely to overreact to highly salient events compared to the low-performing judges.
The Imposition of Pretrial Conditions on Released Federal Defendants: The Overuse of Conditions Without Providing Any Measurable Benefits
Thomas Cohen & William Hicks
Criminal Justice and Behavior, forthcoming
Abstract:
In the federal system, defendants placed on pretrial release may have multiple conditions imposed on them which are aimed at ensuring court appearances and maintaining public safety. In general, little is known about the number of conditions imposed on released federal defendants, the extent to which conditions are associated with pretrial risk, and the potential of these conditions to maximize court appearances and minimize pretrial crime. This study seeks to address these issues by examining the imposition of pretrial conditions on 223,260 released federal defendants. The results show that defendants received an average of about nine conditions and that the association between conditions and a defendant's pretrial risk classification was relatively modest. The results also show conditions having no significant relationship with reductions in the likelihood of pretrial crime or missed court appearances; however, the probability of being revoked increases with the number of conditions imposed.
Legal Pluralism, Arbitration, and State Formation: The Rise and Fall of Philadelphia's Quaker Court, 1682-1772
Esther Sahle
Law and History Review, forthcoming
Abstract:
Legal centralization in British America was characterized by the passing of arbitration from the community level to the colonial courts. As a consequence, when the 1765 Stamp Act raised the cost of court business, colonists were at a loss for alternatives. This paper addresses the question of why, at this point, colonists did not return to earlier, non-state forms of arbitration. It offers an explanation by providing a detailed empirical study of an alternative American legal forum: the Philadelphia Quaker monthly meeting. While busy arbitrating disputes in the early colonial period, it declined from around 1720. Contrary to what might be expected, this decline was not the consequence of state efforts to marginalize competing institutions. Rather, the local Quaker population abandoned their community legal forum in favor of the public courts. This was likely due to the Quaker court's reliance on reputation-based instruments for enforcement. As Philadelphia's population grew, the meeting's practice of pressuring culprits into compliance through public shaming lost its edge. Accordingly, Friends moved their legal business to the public courts. The paper contributes to the debates on the legal pluralism of empires, the history of arbitration, and state formation in the Atlantic.
Interpretable algorithmic forensics
Brandon Garrett & Cynthia Rudin
Proceedings of the National Academy of Sciences, 10 October 2023
Abstract:
One of the most troubling trends in criminal investigations is the growing use of "black box" technology, in which law enforcement rely on artificial intelligence (AI) models or algorithms that are either too complex for people to understand or they simply conceal how it functions. In criminal cases, black box systems have proliferated in forensic areas such as DNA mixture interpretation, facial recognition, and recidivism risk assessments. The champions and critics of AI argue, mistakenly, that we face a catch 22: While black box AI is not understandable by people, they assume that it produces more accurate forensic evidence. In this Article, we question this assertion, which has so powerfully affected judges, policymakers, and academics. We describe a mature body of computer science research showing how "glass box" AI -- designed to be interpretable -- can be more accurate than black box alternatives. Indeed, black box AI performs predictably worse in settings like the criminal system. Debunking the black box performance myth has implications for forensic evidence, constitutional criminal procedure rights, and legislative policy. Absent some compelling -- or even credible -- government interest in keeping AI as a black box, and given the constitutional rights and public safety interests at stake, we argue that a substantial burden rests on the government to justify black box AI in criminal cases. We conclude by calling for judicial rulings and legislation to safeguard a right to interpretable forensic AI.
Are Lawyers' Case Selection Decisions Biased? A Field Experiment on Access to Justice
Jens Frankenreiter & Michael Livermore
Journal of Legal Studies, June 2023, Pages 273-304
Abstract:
The attorney-client relationship is pivotal in providing access to courts. This paper presents results from a large-scale field experiment exploring how demographic information (encoded in potential clients' names) affects how attorneys respond to initial inquiries in private injury cases. On the basis of prior literature, we hypothesize that race is a significant factor, but we also explore race and gender interactions. We find that ostensibly Black or Hispanic inquirers receive fewer responses than ostensibly White inquirers, a result largely driven by preferential treatment of White female inquirers. The racial disparities are larger than those previously documented in contexts such as public services but smaller than in contexts such as employment. We also find suggestive evidence that White attorneys are more likely than others to treat White inquirers preferentially, which implies that the differences in response rates are not merely a reaction to jurisdiction-level factors affecting lawsuits' expected payoffs.
The Only Thing Constant is Change: Temporal Analyses of Racial/Ethnic Sentencing Disparities
Bryan Holmes & Ben Feldmeyer
American Journal of Criminal Justice, October 2023, Pages 1080-1104
Abstract:
Over the last several decades, federal courts have devoted considerable effort towards improving fairness in sentencing. Despite these efforts, research has consistently shown that racial/ethnic minority defendants receive harsher sentences than similarly situated White defendants. While a large body of research has detected these racial/ethnic disparities, relatively few studies have examined how they have changed over time in light of the different legal, societal, and priority changes in federal criminal courts (and the United States more broadly). Using 22 years of federal sentencing data, the current study assesses trends in Black-White and Hispanic-White sentencing disparities (net of factors relevant to sentencing). Results suggest that trends in racial/ethnic sentencing disparities differ by the dependent variable examined. At incarceration, Black and Hispanic disadvantages have been largely time stable. However, racial/ethnic effects on sentence length have changed over time, diminishing in early years before re-aggravating in later years. Findings suggest that the movement towards racial/ethnic equity in sentencing has been slower than many might hope, with setbacks along the way. The re-emergence of racial/ethnic disadvantages indicates the need for a renewed focus towards reducing racial/ethnic disparities and creating greater egalitarianism in sentencing.
Where is the Evidence? Comparing the Effects of Evidence Strength and Demographic Characteristics on Plea Discounts
Kevin Petersen, Allison Redlich & David Wilson
Journal of Quantitative Criminology, December 2023, Pages 919-949
Methods: This study uses a penalized ridge regression to predict counterfactual trial sentences for a sample of defendants who pled guilty. Plea discounts are estimated using each defendant's predicted trial sentence and observed plea sentence. Discount estimates are then regressed on variables related to case evidence and the demographic characteristics of the defendant.
Results: Results suggest that increases in the amount of evidence associated with a case lead to decreases in the size of the plea discount. Both main and interaction effects are observed for race/ethnicity and sex, with Hispanic and male defendants receiving significantly smaller discounts than White or female defendants. Calculation of standardized effect sizes further indicates that demographic characteristics exert larger effects on plea discount estimates than evidentiary variables.
Telling us less than what they know: Expert inconclusive reports conceal exculpatory evidence in forensic cartridge-case comparisons
Andrew Smith & Gary Wells
Journal of Applied Research in Memory and Cognition, forthcoming
Abstract:
Cartridge-case comparison experts are tasked with determining whether a reference sample associated with the suspect originated from the same source as the evidence sample found at the crime scene. Same-source reports have strong inculpatory value and different-source reports have strong exculpatory value. Surprisingly, inconclusive reports have strong exculpatory value as they occur much more frequently for actual nonmatches than for actual matches. We used a signal detection model to test our hypothesis that some experts report inconclusive when they detect a nonmatch. Consistent with this hypothesis, we found that even when examiners were able to perfectly discriminate matches from nonmatches, they rendered inconclusive reports on 32% of nonmatch trials. Experts are biased to avoid rendering different-source reports, which conceals exculpatory information that innocent persons desperately need to establish their innocence. We argue that this biased examiner might result from an adversarial allegiance bias combined with a flawed response scale.
Laffer's Day in Court: The Revenue Effects of Criminal Justice Fees and Fines
Samuel Norris & Evan Rose
NBER Working Paper, October 2023
Abstract:
Many jurisdictions levy sizable fines and fees (legal financial obligations, or LFOs) on criminal defendants. Proponents argue LFOs are a "tax on crime" that funds courts and provides deterrence; opponents argue they do neither. We examine the fiscal implications of lowering LFOs. Incentives to default generate a "Laffer" curve with revenue eventually decreasing in LFOs. Using detailed administrative data, however, we find few defendants demonstrably on the right-hand side of the curve. Those who are tend to be poor, Black, and charged with felonies. As a result, decreasing LFOs for the average defendant would come at substantial cost to governments.
Defamation with Bayesian Audiences
Yonathan Arbel & Murat Mungan
Journal of Legal Studies, June 2023, Pages 445-483
Abstract:
How strictly should the law regulate false defamatory statements? We first show that the presence of judicial errors often puts defamation law on a Laffer curve: regulation that is too lax or too strict is inferior to moderate regulation. While moderate regulation is ideal, it is not always attainable because of practical and legal constraints. With these constraints, we consider a Bayesian audience that takes the strictness of defamation law into account when evaluating statements. The optimal standard is then laxer than is prescribed by standard models with naive audiences. These findings underscore the importance of accounting for audience effects in analyzing defamation law.
The role of cable news hosts in public support for Supreme Court decisions
Scott Simon Boddery et al.
Journal of Empirical Legal Studies, forthcoming
Abstract:
In the current media environment, Americans increasingly tune into cable news programs with distinct ideological brands. This paper extends existing work on media source cues to coverage of the US Supreme Court, an institution which depends entirely on media outlets to communicate its rulings to the American public. We argue that the source cues associated with celebrity media personalities serve as a heuristic that helps individuals form their opinions about public policy. Using a nationwide survey experiment with over 2000 respondents, we find that commentary on Supreme Court decisions from cable news hosts affects public agreement with the Court's rulings, with key differences between how liberal and conservative respondents respond under certain conditions. While unexpected positions espoused by in-group messengers shift the views of liberals and conservatives alike, signals from out-group messengers yield more of an effect for conservatives than for liberals. Our results show that counter-stereotypical (unexpected) position taking has a powerful impact on public perceptions of policy outcomes and suggest that well-known media figures may have an important role in mitigating ideological polarization in America.
The Direct Effect of Corporate Law on Entrepreneurship
Jorge Guzman
Journal of Law, Economics, and Organization, forthcoming
Abstract:
From 1946 to 1983, US states modernized their corporate law by adopting the Model Business Corporation Act (MBCA), a compendium of legal best practices. Better corporate law increased entrepreneurship. After the adoption of the MBCA, the number of new local corporations increased by 26% on average, half of which was substitution from other firm types, and the rest was net-new firms. States that only partially adopted saw no benefit, and the largest increases were concentrated in regions with ex ante lower quality law. At the individual level, people in states adopting the MBCA also report higher self-employment levels, but not higher wage employment or labor force participation. Consistent with the MBCA increasing efficiency and decreasing regulatory capture, the effect was larger for women, black, and those located outside the central city.