Thomas Unise
A disturbing trend has emerged on TikTok where AI-generated videos depicting inappropriate ‘toys’ for children are circulating widely. These videos, created using OpenAI’s Sora 2 video generator, blur the lines between parody and harmful content potentially targeting minors.
The Concerning Trend of AI-Generated ‘Toy’ Videos
Within just one week of Sora 2’s limited release, TikTok’s For You Page was flooded with AI-generated videos featuring suggestive children’s ‘toys’ with clear adult connotations. One notable example is the ‘Vibro Rose,’ a pen-like object with design elements resembling a sex toy, shown being used by photorealistic AI-generated children.
Other videos feature similar themes: rose or mushroom-shaped water toys and cake decorators that dispense ‘sticky milk’ or ‘white foam,’ all being marketed to AI-generated children in commercial-style videos. While not explicitly pornographic, these videos contain obvious sexual innuendos that viewers quickly recognize as inappropriate.
Legal Gray Areas and Growing Concerns
The legal framework surrounding AI-generated fetish content involving minors remains unclear. The Internet Watch Foundation in the UK reports that AI-generated child sexual abuse material (CSAM) cases have doubled in just one year, with most depicting girls.
In response, the UK is introducing amendments to its Crime and Policing Bill to allow ‘authorized testers’ to verify AI tools aren’t capable of generating CSAM. Similarly, 45 US states have implemented laws criminalizing AI-generated CSAM in recent years.
Platform Responses and Moderation Challenges
OpenAI has implemented safeguards to prevent deepfakes of young people, banned CSAM outright, and reports violations to the National Center for Missing and Exploited Children. However, creators are finding ways to circumvent these protections.
The videos exist in a troubling gray area—not hardcore pornography or targeting real children, but designed with apparent intent to appeal to predators when combined with suggestive captions and descriptions.
TikTok has removed many flagged videos and banned accounts uploading this content, but the contextual nuance of these videos presents significant moderation challenges. Content that isn’t explicitly pornographic but subtly caters to fetishes involving children often evades automated detection.
Beyond Sexual Content: The Broader Problem
The issue extends beyond overtly sexual themes. Other disturbing AI-generated videos include fake toy commercials referencing convicted sex criminals, 9/11, Princess Diana’s death, and characters that appeal to inflation, pregnancy, and obesity fetishes—often featuring AI-generated minors.
Experts like Mike Stabile from the Free Speech Coalition argue that AI companies must implement more nuanced moderation practices similar to those used by adult sites, including banning certain terms associated with fetish content and improving moderation teams.
The Path Forward
The Internet Watch Foundation urges platforms to prioritize safety by design, preventing harmful content from being created in the first place. As AI technology advances, the challenge of detecting and removing inappropriate content targeting children will require increasingly sophisticated approaches and possibly new regulatory frameworks.
After WIRED’s investigation and request for comment, OpenAI banned several accounts creating these videos, and TikTok removed many of the flagged videos, though some remain online.
GIPHY App Key not set. Please check settings