New York Police Department (NYPD) officers spent thousands of hours between 2020 and 2023 responding to unconfirmed gunshot detection alerts from ShotSpotter, according to an audit released Thursday.

“ShotSpotter claims to be a ‘proven detection system’ that is ‘faster and more accurate,’ but our audit found that 87 percent of the time, ShotSpotter is sending NYPD officers in response to loud noises that don’t turn out to be confirmed shootings,” said New York City Comptroller Brad Lander in a statement. “The evidence shows that NYPD is wasting precious time and money on this technology and needs to do a better job managing its resources. Chasing down car backfires and construction noise does not make us safer.”

A spokesperson for SoundThinking, ShotSpotter’s manufacturer, shared a statement with Security Management that the company is continuing to review the audit. But at a high level, the spokesperson said the company believes the audit is “gravely misinformed in its assessment of data and the value of ShotSpotter as a critical public safety tool.”

The spokesperson added that “assuming that the lack of confirmation of a shooting automatically means that some other loud event triggered a false alert is erroneous and a perilous assumption. ShotSpotter is a digital witness to criminal gunfire. Perpetrators don’t remain at incident scenes, and police know that they’ll frequently clean up shell casings. The NYPD is a sophisticated law enforcement agency, and they wouldn’t waste precious time and dollars on a system that rendered mostly false positives. The comptroller’s report is wrong on this point.”

More than 170 cities currently use ShotSpotter technology to detect and alert law enforcement to gunfire instances. The NYPD first contracted to use ShotSpotter in 2014 and deployed the system in 2015. So far, it has spent $45 million on the product with an additional $9 million contracted through 2024, according to the audit. The NYPD’s ShotSpotter system uses more than 2,000 acoustic sensors across the city to detect gunshot activity.

Where sensors are placed is determined by data, including historical information on shootings and homicide rates, the SoundThinking spokesperson said. The exact location of these sensors is not disclosed to clients, including the NYPD, to protect community members and residential and business owners who volunteer to place the sensors on their buildings.

“In addition, there is a risk of tampering or vandalism to sensors if the locations are made public,” the spokesperson explained.

Under the existing NYPD contract, ShotSpotter is expected to report 90 percent—or more—of probable shootings, where outdoor gunfire incidents that use commercially available rounds greater than .25 caliber will be detected and located within 25 meters of the gunshot location.

Lander’s office began an audit of the NYPD’s use of ShotSpotter on 1 July 2020 that lasted through 30 June 2023. The audit found that the existing contract performance standard the NYPD uses does “not measure whether alerts sent to NYPD result in confirmed shootings,” and results in “artificially high ratings for ShotSpotter.”

Confirmed shootings are defined as incidents where the NYPD recovers evidence, firearms, ballistics, video, eyewitness accounts, victims shot, summary arrests, or 911 calls that report a shooting.

When a ShotSpotter sensor detects a suspected gunshot, it sends the alert to an Incident Review Center staffed by ShotSpotter employees who either confirm or change the incident type classification—a process designed to take 60 seconds or less. ShotSpotter uses technology and science to filter out sounds that are likely not to be gunfire—such as bangs, booms, or pops—during this process.

“ShotSpotter technology has a 97 percent accuracy rate for detections across all police department customers nationwide for the last four years,” the spokesperson said. “This statistic has been independently verified by data analytics firm Edgeworth Analytics.”

If Incident Review Center staffers confirm that an incident is a gunshot, the alert is sent to the NYPD to respond.

“For the eight months in 2022 and 2023 reviewed by the audit, these unconfirmed shootings ranged from 80 percent to 92 percent of ShotSpotter alerts, sending officers to investigate 7,262 incidents that did not turn out to be confirmed shootings,” according to the comptroller.

Auditors’ analysis found that NYPD officers spend an average of 20 minutes investigating unfounded alerts and 32 minutes investigating unconfirmed shots.

“This represents a total of 940 minutes spent investigating unfounded alerts and almost 25,000 minutes investigating unconfirmed alerts in a single month,” the audit said. “The two categories combined represent 426.9 hours during a single month. If a single officer responded, that equates to 36 twelve-hour shifts; if two officers responded, the number of wasted shifts doubles.”

When asked by auditors about the findings, the NYPD said that confirmed shootings is not an appropriate way to measure performance because it’s based on investigation at the scene and does not include later events. Currently, however, the NYPD does not collect data that could be used to test this assertion.

“NYPD does not track if a shot that is unconfirmed in the immediate aftermath of an alert is later confirmed as a shooting, but this is not a good reason not to measure ShotSpotter based on the detection rate of confirmed shootings,” the audit explained. “NYPD should capture and evaluate all information necessary to accurately determine the detection rate of confirmed shootings.

“Knowing whether ShotSpotter accurately detects shootings is an essential piece of information needed to determine whether it is an effective tool,” it continued.

Auditors also wrote that both ShotSpotter and the NYPD “significantly overstate” reduction in response times that ShotSpotter technology facilities. They discovered the discrepancy by analyzing ShotSpotter alert response times in comparison to 911 call response times to shots fired outside.

Auditors found that “during the month of June 2023 average response times to ShotSpotter alerts were 1 minute and 38 seconds faster than response times to reports to 911 of outdoor shots fired (3 minutes 50 seconds for ShotSpotter, versus 5 minutes 28 seconds for 911), far less than the difference of 5 minutes claimed in publicly available data.”

In a statement shared with Security Management, a spokesperson for the Office of the Deputy Commissioner, Public Information, which handles media requests for the NYPD, declined to answer questions about the audit’s findings and said the NYPD “consistently reviews the effectiveness of technologies it utilizes to combat crime. ShotSpotter remains an integral tool in the NYPD’s mission of addressing gun violence and keeping the public safe.”

The auditors made five recommendations to the NYPD to address their findings; the department accepted two and rejected three.

One recommendation that the NYPD rejected was to decline to renew the ShotSpotter contract when it expires in December 2024 unless a thorough performance evaluation—including low rates of confirmed shooting incidents detected and NYPD officer time spent responding to alerts—was conducted.

In the NYPD response to auditors included in the final audit report, Deputy Commissioner, Management and Budget, Deirdre J. Snyder wrote that it was “not feasible to conduct an evaluation prior to the contract renewal period.” The department also said that the loss of the ShotSpotter program would “result in a less safe working environment for officers and an increased chance of violent encounters for all New Yorkers.”

The NYPD also rejected a recommendation to create a more meaningful performance standard for ShotSpotter’s accuracy to factor in the high rate of alerts not confirmed as shootings. Snyder wrote that the NYPD is limited in what it can consider a “confirmed shooting” with a ShotSpotter alert because of the nature of police work and alerts that do not result in the recovery of evidence.

The NYPD pushed back on auditors’ recommendation to collect and publish relevant ShotSpotter data to increase transparency, and declined to answer Security Management’s questions about releasing data in format that is easy for the public to understand.

“Shooting numbers are ever changing based on investigations and other factors that follow the shootings and therefore, parties that are not familiar with the data could misinterpret the information,” Snyder explained. “In addition, this would need to go through various channels to determine feasibility and legality of such an information request.”

The NYPD did confirm, however, that it will continue to follow up with ShotSpotter on areas where the 90 percent performance rate is not met. The department cited its past work with the manufacturer to improve performance in Manhattan as a benchmark for these discussions.

Between August 2019 and December 2022, for instance, auditors found that ShotSpotter failed to meet the 90 percent performance rate in Manhattan in 30 out of 41 months. ShotSpotter told auditors that the reasons for this included high levels of noise in Manhattan, construction in Harlem, and density of buildings in the borough. The company has since moved and replaced sensors in Harlem, and between January and June 2023 the borough met the 90 percent threshold for performance.

Lander said the NYPD’s response to the audit findings was “disappointing and reflects a disinterest in using data, effective performance metrics, and transparency to improve public safety. With a thorough evaluation before deciding whether to renew this multi-million-dollar contract, better performance standards, and more transparency, the NYPD could deploy its resources—especially its officers’ time—far more effectively.”

 

For more on audio detection, response, and communication, check out our May series on the topic.