Flock Safety’s latest search tools give police several ways to find people on camera, not just the license plates on their cars. The capabilities sit inside a suite of software now at the center of dozens of police stalking cases. WIRED collected and analyzed that software and created a mock-up of what officers see when interacting with the system.
Among the tools is an AI-powered watch list. An officer can cordon off an area of a city and make the cameras inside it run a continuous, automated search for anyone fitting a written description.
Over the past year, Flock has faced backlash against its technology. Police departments across the country have canceled contracts, and city councils that once approved Flock have quietly voted the cameras back out. Two members of Congress wrote to Flock CEO Garrett Langley after a Texas deputy searched more than 83,000 cameras for a woman who had an abortion. And Illinois found the company had let federal immigration agents access state camera data in violation of state law.
Flock answered in August with a package of changes that shortens the default period over which data are retained, requires case codes for every search, and adds automated auditing to catch officers misusing the tool—all of which Flock says will be mandatory by the end of the year. WIRED examined the code those promises run on, but much of what governs a search happens on Flock’s servers. Crucially, the analysis found that Flock’s guardrails may discourage some abusive uses of its tools, and will record them, but will not stop them.
Experts who study automated screening and police technology say Flock has given its AI a job no technology does reliably. Flock’s system decides, out of sight of the department using it, which descriptions of a person an officer may search for—and no screen operating at that scale is accurate, the experts say. Further, nobody outside the company can measure how often it is wrong, or in which direction. This puts users in a bind: The company itself warns officers that results may be inaccurate or incomplete and should not be relied on alone but puts the “risk of any inaccuracies” on the officer running the search.
Some of those decisions experts find harder to defend than others. Flock’s system scores an officer’s written description against several categories of sensitive content, including race and religion, and “political, social, and cultural expression” is the only one that will not stop a search.
Tom Bowman, policy counsel with the Center for Democracy and Technology's security surveillance project, says the one sensitive category Flock treats with a speed bump instead of a stop is the one courts have guarded most closely. Political and cultural expression is “one of the most protective areas” under the First Amendment, he says, and it “seems like that actually has the least concern from a policing perspective.”
WIRED sent Flock a detailed description of each of its findings along with 16 questions. The company responded with a written statement saying its search tools are built to help police find relevant information “while maintaining clear safeguards around how the technology can be used.” Queries are evaluated against its content policies, it says. The statement does not describe how its AI model was built, what instructions it works from, or what separates one verdict from another.
Data flows in both directions. When the software flags a search, it sends Flock the language the officer used, the category and confidence score the model assigned it, and whatever the officer did next, whether continuing, dismissing, or disputing the warning. Flock’s statement does not address how long those records are kept, who inside the company can see them, and whether they are used to evaluate or improve the moderation system.
Every Vote Counts
Flock’s interface holds several tools, and each one works differently. A search that uses text to find images, for example, produces a statistical ranking of which footage best fits the words. Every image a camera captures is stored as a set of numbers describing what it shows, and any description typed into the search box is converted into numbers the same way. A model then compares the two, ranking the nearest footage at the top. Anything that might otherwise influence the rankings—additional filters, thresholds, or instructions fed to Flock’s model—remains out of view on the company’s servers.
A feature called Smart Sort lets the officer approve or reject each image that comes back. A few votes rearrange the ranking, moving footage that resembles the approved images to the top. “Too many results?” a tooltip asks. “Vote on a few results to quickly refine.”
An officer looking for a vehicle picks from menus of colors, body types, makes, models, and features like roof racks and bumper stickers; a person search, by contrast, includes no filtering and accepts only a written description. An example offered by the system: “person wearing scrubs.” Flock calls this plain-language search FreeForm and says it has sold it for years. 404 Media reported in July that officers were using FreeForm to search for people using descriptions of clothing and tattoos.
Standard vehicle search and FreeForm are separate capabilities, Flock says, with standard search running on defined vehicle attributes and FreeForm on natural-language descriptions. On video cameras, the company says, FreeForm searches “clothing, colors, objects, or other case-relevant details.” On license plate cameras, it is limited to vehicles, and person attributes cannot be searched there at all.
The code, which WIRED collected using custom software that gathers files Flock sends to a user’s browser before they log in, sets out which searches officers can run, which controls and warnings they see, and how one result becomes the basis for the next. Out of reach on company servers are any actual search results, camera feeds, or department settings, along with the AI model that answers police prompts.
The code WIRED collected holds Flock’s user interface and shows both the network calls it makes and the answers it expects in return. It shows that departments can, for now, configure the tool for some searches without a reason or a case number linked to a real investigation and can leave off the automated system that flags suspicious use; Flock says the controls will be mandatory by year’s end.
Search activity is recorded in audit logs, Flock says, and required search restrictions apply across its search tools. The company did not say whether anything on its servers would stop a search submitted without a reason or a case number.
Fired, Suspended, Resigned
Police officers have already used Flock’s software to track people for reasons unrelated to police work. At least 50 officers in the US have recently been charged with or accused of misusing license plate readers, according to The Washington Post. Forty-six of those cases involve Flock. In more than half, the alleged target was a wife, a girlfriend, an ex-partner, an ex-partner’s new boyfriend, or a woman the officer wanted to meet.
Details of how police databases can be abused appear repeatedly in the reports of officer misconduct: A Milwaukee officer searched for a woman he was dating 124 times and her former partner 55 times, and logged each search as an “investigation.” A Georgia police chief ran his ex-girlfriend and her daughter 600 times, was charged, and took his own life five months later. A Kansas chief, who was never charged, ran his ex 164 times and her boyfriend 64 times, and lost his badge. A North Carolina officer ran her boyfriend's ex-wife 31 times and filed 29 of them as traffic infractions.
Police abusing investigative tools is not new. A 2016 Associated Press investigation found officers running database searches on exes and people they found attractive, with police employees at state agencies and dozens of the nation’s largest departments fired, suspended, or forced to resign, more than 325 times from 2013 through 2015. WIRED reported in 2023 that Immigration and Customs Enforcement employees and contractors had been investigated at least 414 times since 2016 for misusing sensitive government databases and found hundreds more allegations against Customs and Border Protection employees this year, among them officers who queried databases to track down women they wanted to date, monitor family members, and follow colleagues’ phones.
Two notification systems in Flock’s software are aimed at human beings. The watch list based on police search queries is one. It runs on an officer’s own written description: They draw a shape on a city map, the alert runs against every camera inside it, and anyone the model decides is a match triggers a notification. The frequency of alerts can be adjusted.
The other, dubbed a People Detection Alert, takes no description at all. Here an officer draws a box inside a single camera's field of view—a doorway, a stretch of pavement—and is notified when a person is inside it. One person is enough to trigger it. The software will not accept an alert set below 75 percent certainty that what the camera is looking at is a person.
Three Verdicts
Flock described its moderation system in April, saying it was designed to ensure First Amendment protections are “baked in.” The code WIRED reconstructed shows an AI model on Flock's servers scoring an officer’s description of a person against eight categories of sensitive content before it runs against stored footage, then returning one of three verdicts: allow, block, or warn. The model relies on a confidence score to reach that verdict. Because the model runs on Flock’s servers, the department never sees that score. All it gets is a canned warning or block.
Blocked categories of search include race or ethnicity, religion, nationality, “subjective or biased terms,” and “offensive content,” which the code describes as sexual content, innuendo, offensive words and phrases, and slurs. In vehicle mode, the tool blocks any search using person-related terms, including gender, clothing, and behavior.
Religion and nationality are the only two categories written both ways, with wording for a block and wording for a warning that lets the officer proceed. Political, social, and cultural expression is written only as a warning.
Flock tells WIRED that officers cannot run searches using prohibited attributes, including religion and nationality, and that an attempt to search prohibited terms “will be blocked.” Asked what circumstances produce a warning in those two categories instead, the company did not say.
When a T-shirt or a bumper sticker draws a warning because the text includes speech with “constitutional protections,” officers are told the search will be logged and that administrators will be notified. Officers then have to tick an acknowledgment box and leave a comment before they’re free to click Continue With Search Anyway.
Flock says the warning leaves room for legitimate work, offering as an example a victim who describes a suspect in a biker-gang jacket bearing “a certain gang logo or emblem containing a flag or other insignia.” An officer who proceeds is reported to an administrator at their own department for review.
Kate Ruane, who directs the Center for Democracy and Technology's free expression project, has spent years studying how automated moderation systems work. Political, social, and cultural expression is “an incredibly amorphous category,” she says, and one reason an officer’s search might contain language including it is to see who attended a protest. That's already happened. The category worries Flock enough that the system returns a warning, she points out, “but you can still get the results if you just click through.”
“No content moderation done at scale is necessarily accurate,” she says. Analysis of moving video, she says, is harder still, and it goes wrong more often.
The categories that do get blocked can be worked around, Ruane says. The system blocks searches based on religion but not clothing, meaning that in practice an officer could get around the block by searching for the distinctive dress worn by some members of a particular faith. “Lots of people are having their images returned in response to these types of queries that would probably be upset if they knew about it.”
In a search last year, an officer in California typed “American flag.” The search drew a block when aimed at a person, then ran across 11,000 cameras when aimed at vehicles instead.
A warning stops only those officers not already determined to run the search, says Deepak Kumar, an assistant professor of computer science and engineering at UC San Diego. Kumar, who studies trust and safety systems, points to interfaces built to address online harassment, where motivated users sometimes did more harm after being warned, and to browser alerts meant to steer people away from malicious websites, where the effect depends on how warnings are presented.
“This kind of logging can be useful administratively, to see which officers or end users bypass warnings often, but that value depends entirely on whether anyone administrates it,” Kumar says. “It functions less as a deterrent and more as a record.”
Reviewing search terms before a query runs will inevitably catch many attempts at misuse, but in isolation, such safeguards are known to fail without equal scrutiny of what the model sends back. “Best practice examines both inputs and outputs,” Kumar says, “which is why so many AI safety efforts now check both to prevent models from generating harmful outputs.”
Jay Stanley, a senior policy analyst at the ACLU's Speech, Privacy, and Technology Project, agrees that the moderation system addresses a narrow slice of the problem. The filtering, he says, is “a speck on the back of a giant surveillance machine” whose database can be probed by AI for “your-imagination-is-the-limit fishing expeditions.” Nobody outside Flock can say how its statistical models reach those calls.
Like Flock’s other recent reforms, he says, the moderation appears to be aimed at “bad-apple cops” and leaves the structural dangers of mass surveillance untouched.
Local Rules
The searches described so far are run against every person or vehicle Flock’s cameras have seen, back as far as an agency stores footage. Besides a written description, the tool takes a plate or a cropped photograph, returns information about cars present at locations that an officer marks on a map, and infers associations from driving patterns, handing police who enter a plate number the vehicles seen beside it. How far a search reaches depends on the search mode, the agency's permissions, and the agreements it holds with other organizations. A search can run on the agency’s own cameras, on cameras other organizations share, or, for some plate searches, across statewide and nationwide networks.
Flock says much of the responsibility for governing its technology sits with local agencies, which set their own rules for access and use. Don De Lucca, a former Miami Beach, Florida, police chief and past president of the International Association of Chiefs of Police, says that is largely right. Vendors can recommend safeguards, he says, but departments have to set the rules before the technology goes into use. “It has to be tied to some kind of legal reason.”
A police commander who oversees special investigations at a Southern California police department, and who requested anonymity because he’s not authorized to speak, says the possibility of abuse is not unique to Flock or surveillance technology. New technologies routinely create opportunities for both legitimate use and for misuse, he says. The real question is “whether the technology gets implemented with strong guardrails, active oversight, auditability, and consequences for any misuse.”
As scrutiny of Flock intensified this spring, the company announced Audit Assistance, which surfaces patterns that may warrant review. Flock says more than a third of its customers enabled it in August. The code WIRED reviewed shows it currently off by default. It can reportedly flag repeated searches by one officer for the same plate, searches limited to other agencies’ cameras, and for one plate searched under multiple case numbers.
Flock says account lockouts for abnormal activity will arrive alongside it. A South Carolina sheriff's office turned Audit Assistance on earlier this month, and an internal-affairs officer identified more than 2,700 allegedly unauthorized searches the next day.
Comments
Back to top