And most recently, in October, an anonymous New Jersey high school student filed the well-publicized lawsuit that has garnered national attention. ClothOff, an app developed for producing deepfake pornography, is the lawsuit’s main target. This troubling technology has reportedly been terrorizing young women online for more than two years, as it allows users to alter images and videos without consent, often creating harmful and illegal content.
The plaintiff was only 14 years old when the original photos the plaintiff posted on Instagram were taken. Through interviews, she learned that her peers had used ClothOff to manipulate those photos. The law therefore treats the AI-generated copies of her photos as child abuse images. This proves the devastating impacts of this technology on children.
Despite ClothOff being removed from major app stores and banned from most social media platforms, it remains accessible through the web and a Telegram bot, raising concerns about the ongoing threat posed by this application. Ultimately, the suit is looking to force ClothOff’s owners to remove all currently posted images and shut them down for good.
Legal scholar John Langford pointed out that it was—and is—up to tech companies to ensure such abuses don’t occur, and the abuses aren’t inevitable. He stated, “ClothOff is designed and marketed specifically as a deepfake pornography image and video generator.”
The history of ClothOff dates back to when it was incorporated in the British Virgin Islands. According to the latest intelligence from the European Union’s anti-money laundering agency, a brother and sister team out of Belarus operate the platform. This new international angle makes it increasingly difficult to see these operators held accountable in U.S. courts, especially in cases of minors.
Besides ClothOff, other technologies such as xAI’s Grok chatbot have been criticized for creating non-consensual pornography. Other countries, such as Indonesia and Malaysia, have already moved to restrict access to Grok over these fears. In parallel, regulators in the United Kingdom have launched an investigation into Grok that could result in similar prohibitions.
Langford told us that he is appalled by the current lack of oversight around these new technologies. He remarked, “How can you not have had more stringent controls in place to make sure this doesn’t happen? That is a kind of recklessness or knowledge but it’s just a more complicated case.”
The ramifications of this lawsuit go beyond the individual lawsuit. First, it brings urgent questions to bear regarding accountability for bad actors in the tech industry, especially given the impotence of the current regulatory framework to combat child exploitation. Langford keyed in on something very important. Last I checked, per the First Amendment, Child Sexual Abuse material is no doubt illegal unprotected expression.
Langford pointed out the legal ramifications for individuals involved in distributing such material: “If you are posting, distributing, disseminating Child Sexual Abuse material, you are violating criminal prohibitions and can be held accountable.”
This lawsuit is still developing, but it has the potential to be an extremely important test case. Most importantly, it will confront the dangers and abuses of deepfake technology directly. This case serves as a critical reminder for stronger protections. It seeks to remind everyone—including lawmakers—of the need to take proactive, not just reactive steps to protect our most vulnerable from online exploitation.
