Editor’s note: This article contains descriptions of imagery depicting child sexual abuse. Reader discretion is strongly advised.
Early last month, Meta deleted around 50 ads posted on Facebook, Instagram, and Threads that directly included child sexual abuse material (CSAM), following reporting by WIRED. At the time, the company said that most of the abusive ads had been published prior to it rolling out new AI tools designed to “better detect and block” harmful ads being uploaded. That AI system, however, doesn’t appear to have been entirely effective.
Instead of spotting a decline in the abusive ads in recent weeks, researchers say they have now discovered hundreds more of the disturbing ads containing graphic sexual scenes involving children—including some that were identical to those Meta recently removed. More than 250 additional ads containing CSAM have run across Meta’s social media platforms since the start of August, according to researchers at the nonprofit Tech Transparency Project who shared its findings with WIRED. Unlike the 53 ads TTP researchers initially discovered, they were able to identify images of real children in the new ads.
These included sexual content depicting a minor belonging to a European royal family, transforming an officially released photograph into a video of a “graphic sex act.” The researchers are not naming the royal to protect the victim and say they contacted officials in the country. WIRED confirmed the source image came from the official website of a royal family.
“What we’re talking about here is not simply AI-generated child sex abuse material,” says Katie Paul, the director of the TTP. The researcher says it appears some images have been taken from the web before being used in the ads. “There’s a lot of harm that can come from this all while Meta is raking in money in the process.”
Overall, the researchers discovered that Meta has published more than 350 of the abusive video ads since the end of last year, with many of them linking to harmful harmful “nudification” apps linked to Chinese developers. In each instance, the researchers immediately reported the ads to US child safety authorities and through Meta’s reporting tools when ads were live and being shown to users at the time of their discovery. The researchers shared detailed findings with WIRED, which independently identified hundreds of other related “nudification” ads containing explicit content of adults on Meta’s platforms.
While the child abuse ads have now been removed for violating Meta’s policies on child exploitation, adult exploitation, and nudity, all of the ads were originally published despite the company’s policies saying it reviews all ads that run on its platforms. In some instances, the TTP researchers say, it took Meta a week to review live ads they had reported for containing CSAM with Meta’s tools. During that time, the ads racked up hundreds more views, according to screenshots seen by WIRED.
When some ads containing child sexual abuse were removed, TTP says, they did not initially include disclosures indicating that they violated Meta’s child sexual abuse policies; those details only appeared to be added after WIRED asked Meta about the discrepancy.
The cache of video ads published across Facebook, Instagram, Messenger, Threads, and Meta’s broader advertising network since the end of last year follow similar patterns. Many include an innocuous picture of a preteen or teenage girl—only one contained a boy, according to the researchers—and overlaid text saying it is “not just a photo” and “there are no restriction” [sic] on what can be done with the image. When the seconds-long video ads start playing, the children’s faces are morphed into graphic sexual videos, often showing them performing sex acts. When clicked, the ads directed people to download AI-face swapping or video apps from Apple and Google’s app stores. “This is the AI that men actually use,” many of the ads say.
Paul says the researchers were able to find the real-world identities of four minors whose images were used in the ads, three of which are from the US. “There were several photos from the same teen who has a social media influencer account on one of Meta’s platforms,” Paul says. The second instance involved another teen influencer who had a public account, the researcher says. “One was a stock photo model that was a real person,” Paul says of the third instance. “It was actually labeled as pre-teen on the stock photo websites.” The final instance was the European royal family member.
The researchers reported all of the ads they found to the National Center for Missing and Exploited Children’s (NCMEC) CyberTipline, which allows people to report online child sexual abuse material, and say they are consulting on how to appropriately disclose their discoveries to the those impacted.
The ads were collectively shown to more than 29,000 accounts in countries across the European Union, plus around 7,000 accounts in the United Kingdom, according to data in Meta’s ad library, which shows live and archived ads, including the harmful ads that were no longer being shown to people. The ads were likely seen by thousands more people than Meta’s ad library indicates, with more than 250 of them appearing in the United States, 120 in Australia, and 80 in India, where Meta does not publish granular ad performance data.
“We don’t tolerate nudify apps or any kind of child exploitation, whether real or AI-generated,” says Meta spokesperson Tracy Clayton. The company has previously removed hundreds of thousands of ads for violating its policies on nudification. The statement claims that “many” of the ads found by the researchers had already been removed and most had fewer than 200 impressions, with the company receiving “under $5,000” in ad payments.
In an email to WIRED, Meta says it reports child exploitation content to NCMEC and is reporting the ads to the organization. Meta minimized the findings by saying that the AI-generated videos featured adults and the clips of the minors were brief or blurred, which made detection of the content more difficult. “The criminals behind this activity know we’ve built strong defenses, including systems that block violating ads as well as ongoing monitoring in case we initially miss something,” Clayton says.
The TTP researchers’ discovery of hundreds of ads containing child abuse material comes amid Meta paying up to $16.7 billion to settle lawsuits over its social networks allegedly harming children. Meta is also appealing a New Mexico court decision that, if enacted, would require it to improve its CSAM reporting processes and include human reviews in the state.
Multiple lawmakers and regulators now say they are investigating or looking into the ads following WIRED’s reporting. “It is disappointing that Meta continues to run ads featuring child sexual abuse material and nonconsensual intimate images after it was brought to their attention,” says Virginia senator Mark Warner, who sent Meta CEO Mark Zuckerberg a letter following WIRED’s initial reporting. The letter has not been replied to yet, according to Warner’s office. “Meta needs to follow their own advertising standards, heed the call, and put an end to these illegal and inappropriate ads,” he says. (Meta tells WIRED it will reply to the letter).
A spokesperson for Michigan’s attorney general says their office “is aware of reports concerning AI‑generated child sexual abuse material appearing in advertising on Meta. This is extremely troubling, and we are actively looking into the matter.” Similarly, Florida attorney general James Uthmeier posted on X that his office is “investigating.”
Meanwhile, a spokesperson for Australia’s eSafety internet regulator says it is “highly concerned” by the reports and that it is assessing Meta’s compliance with its rules. It says it has requested “information on this matter from the company at a senior level.”
Lacking Protections
Over the last six years, dozens of AI nudification or “undress” apps and websites have flourished online, with them being used to harm countless women and girls. As this dangerous nonconsensual intimate imagery (NCII) ecosystem has continued to grow, with those behind it estimated to be earning millions each year, child protection organizations have warned the technology is “increasingly” used to create CSAM.
“AI apps are making it far too easy to create realistic criminal imagery of child sexual abuse, creating the new risk of everyday photos of children being taken and transformed into sexual content,” says Emma Hardy, director of communications at the Internet Watch Foundation, a UK-based nonprofit which removes child abuse content from the web. “Platforms and companies must make sure there is nowhere these apps can be advertised and spread.”
As WIRED previously reported, some of the ads discovered by TTP researchers were posted by Chinese companies that act as ad “resellers” in the country, where Meta’s platforms are banned. Many of the apps the ads linked to also appear to have Chinese connections, according to TTP. (Meta has previously launched legal action against a Hong Kong-based company linked to one set of nudifier platforms).
The ads pointed people towards almost 50 apps combined across Google’s Play Store and Apple’s App Store. Of the 350 ads, the researchers say that around 300 of them directed people to AI image or video generator ads, with most of these apps being linked to China-based developers. Some apps had already been removed by the time WIRED started reporting this story.
Among those that would generate explicit imagery of adults were eight identical apps listed on Apple’s App Store, multiple of which were linked to Chinese developer names. When the apps opened, they would show explicit AI-generated content, including a menu of sexual scenarios, including one called “threatening knife” where a knife is held to a woman’s throat. WIRED identified another seven identical apps that shared the same design, explicit content, and listed the same developer contact details.
“The App Store was designed to be a safe and trusted place for users, and we have zero tolerance for developers who deliberately attempt to evade our review process,” says Apple spokesperson Adam Dema. “The developers mentioned in this report initially submitted apps that are compliant with the App Review Guidelines, only to later surface nudify capabilities, which are against our rules.” The company says that 17 of the 20 apps included in the research had already been removed from its store and it has taken action against the remaining three, as well as removing the seven apps WIRED found.
Google spokesperson Dan Jackson says the company does not allow apps that contain sexual content or the ability to nudify. “We continually take proactive steps to detect and remove apps with harmful content,” they say, adding that it removed all the apps included in the report. “When violations are reported to us, we investigate and take swift action, which in the case of these apps has included suspending hundreds of violating apps and restricting related search terms like ‘nudify’ on our store.”
“The share of Chinese developers creating nudify apps has grown year over year,” says David Pham, a former Meta employee who investigated nonconsensual imagery and abuse on its platforms. Pham says that while at Meta he observed similar ads to those discovered by the researchers and those uploading the content would constantly try to get around safety systems.
“Every time the platform or investigators would build a new detection model or a new way to have some type of enforcement, they quickly—within a day, maybe five days—built a new way to avoid detection,” Pham says. Those uploading the ads would change domains they used, obscure who posted them, and include measures such as blurring imagery or subtly changing images to avoid detection. However, Pham, who left Meta last year, says it has the funding and resources to be able to tackle the uploads. “I would like to live in a world where we’re not getting CSAM or NCII material sent directly to our phones,” he says.
Meta’s current protections against ads containing CSAM and NCII appear to need work. More than 11,000 undress ads have appeared on Meta’s platforms since February this year, according to a recent Indicator Media report, with many of these being reportedly posted by “repeat offenders.” Analysis by WIRED also found hundreds of ads including explicit content that contain similar language to those ads found by the TTP researchers. (Meta says malicious advertisers use adversarial tactics to continually try to avoid its detection systems and it has “incorporated” the language in the ads highlighted by WIRED into its moderation systems.)
One ad containing one of the teenagers that the researchers found the real-world identity of was reported to Meta when it had reached four accounts in Europe, according to the TTPs research. However, when it was removed days later, it had been seen by more than 330 users. “This is not something that it should take a small nonprofit doing research to stumble across and then have to investigate and report,” Paul from TTP says. “We’re talking about one of the most well-resourced— technologically and financially—companies on the planet.”
Comments
Back to top