The Code in the Courtroom: Algorithmic Risk Scores Under Fire in San Francisco Courts

Photo Credit: Electronic Frontier Foundation

(SAN FRANCISCO, CA.) - The recent implementation of algorithmic justice in the San Francisco Bay Area has resulted in a quintessentially modern tension between the age of artificial intelligence and constitutional fairness.

San Francisco officially rolled out an automated risk-assessment software, Public Safety Assessment, designed to determine the recidivism rate of arrested individuals for determining bail, degrees of supervision, and whether individuals should be held in custody prior to their trial.

The integration of this software, developed by private research initiatives, was intended to “standardize” pretrial release recommendations, claiming to remove human biases from the courtroom, with supports claiming the calculated numerical risk scores for a defendant’s likelihood to recidivism would successfully weed out human biases from the courtroom; however, scrutinization from researchers such as UC Berkeley’s Computational Research for Equity in the Legal System (CRELS) and various legal experts have arrived to the same troubling conclusion: the AI program serves as a prime example of systemic bias posing as neutral technology.

An illustration of this ongoing friction between AI risk scoring and constitutional protections was demonstrated in the San Francisco federal court case Simon v. City and County of San Francisco, in which criminal defendants challenged various pretrial release conditions partially determined by the Public Safety Assessment. In this case, individual Bay Area defendants were required to submit to the Sheriff’s Office Pre-Trial Electronic Monitoring Program for the city’s risk evaluation system.

Under these rules, in 2024, individuals were placed on “automated release monitoring,” forcibly subjecting them to agree to warrantless searches and continuous location tracking from law enforcement. Defense attorneys representing the plaintiffs argued that while the Public Safety Assessment was framed as a neutral mechanism to avoid jail time, it actually just traded physical incarceration for invasive algorithmic surveillance, as searches did not require individualized judicial warrants.

The legal challenge in this keystone case underscored how algorithmic risk assessment instruments generate a distinct feedback loop that penalizes individuals long after their initial arrest. When the software calculates a risk score using historical factors such as prior arrests, missed court hearings, and previous convictions, it inevitably reflects existing disparities in neighborhood law enforcement deployment.

Higher scores then lead courts to mandate strict technological supervision, such as ankle-GPS tracking, as a mandatory condition for release. If a defendant incurs a technical violation under these stringent automated monitoring rules—such as entering a restricted zone or missing a check-in due to routine transit delays—that administrative infraction is logged into the court record as a new failure. When the algorithm re-evaluates the individual in future proceedings, this logged event artificially escalates their risk profile, driving up their score and triggering harsher penalties in a cyclic manner.

Photo Credit: California Law Review

While private entities like Arnold Ventures designed the Public Safety Assessment, local court systems hold the authority that governs how scores map these release conditions. Defense counsels representing defendants in the Bay Area courts now face immense hurdles when attempting to inspect the underlying software code to examine how algorithms translate into court surveillance orders.

Since defense teams are routinely blocked and prevented from accessing the mechanized code logic embedded into these AI tools, defendants are subjected to intrusive pretrial motions driven by metrics they cannot scrutinize. The ongoing legal debates surrounding these programs demonstrate incorporating unchallengeable automation systems may pose more of an obstacle to constitutional rights than a removal of biases. 

Correction or Addition? If you would like to add information to this article or suggest a correction, please contact me at alizeh.i@lead4earth.org.