The rise of artificial intelligence has brought many advancements, but it has also birthed a sinister shadow: the “nudification” of real people through deepfake technology. This reality hit a legal turning point this week when San Francisco City Attorney David Chiu issued formal cease-and-desist orders to tech titans Apple and Google. The demand is clear and urgent: purge the app stores of tools that allow users to strip clothing from photos of real people, stop profiting from the commissions generated by these apps, and cease what Chiu describes as “aiding and abetting” the creation of non-consensual sexual content. For too long, these platforms have allowed these predatory apps to slip through the cracks, effectively turning a blind eye while harvesting service fees from tools designed to violate human dignity.
The core of this issue is the staggering ease with which an everyday photo can be transformed into a weaponized image of sexual abuse. Modern AI, often accessible with just a few clicks and a handful of dollars, has democratized the ability for anyone to generate graphic, realistic deepfakes of neighbors, classmates, or strangers. These apps essentially function as digital tools for harassment, disproportionately targeting women and girls. Research has revealed the chilling scale of this problem; there are documented cases across at least 90 schools where students have used these platforms to create and circulate sexualized deepfakes of their peers. When these power-tools for abuse are listed in the same app stores that children use to download games or homework helpers, the platform owners share a moral burden that they can no longer ignore.
City Attorney Chiu’s legal notices argue that by providing the infrastructure—and taking a percentage of in-app purchases—Apple and Google are inextricably linked to the harm these apps cause. He highlights a fundamental failure in moderation: despite claims that these mega-corporations are at the forefront of AI safety, they continue to host software that serves no purpose other than to violate privacy and orchestrate digital sexual violence. By profiting from transaction fees on these downloads, the tech giants are not merely passive storefronts; they are financially incentivized participants in an ecosystem that degrades human rights. The letters serve as a stark warning: California law does not permit companies to hide behind user-agreement loopholes while facilitating services that monetize exploitation.
Historically, Apple and Google have handled these reports with a “whack-a-mole” approach—removing apps only after public outcry or investigative journalism forces their hand. Google has stated that it has already purged hundreds of such apps from its store and restricted search terms like “nudify” to curb access. While these actions indicate a degree of responsiveness, they also acknowledge that the current system is fundamentally reactive rather than preventative. The sheer volume of these apps suggests that the vetting processes currently in place are insufficient at best and willfully negligent at worst. For a person whose image has been stolen and violated, a belated “policy violation” removal is cold comfort after the damage has already occurred.
The human cost of this technology extends far beyond the screens of a smartphone. When an individual’s identity is co-opted for non-consensual sexual content, it can cause lasting psychological trauma, damage reputations, and dismantle a person’s sense of safety in their own community. As AI becomes more sophisticated, the line between reality and fabrication continues to blur, making the legal and ethical arguments against “nudify” apps increasingly undeniable. These are not merely pieces of harmless software; they are digital mechanisms of coercion. By continuing to host them, Apple and Google are deciding that a small percentage of app store revenue is worth more than the safety and privacy of the users they claim to protect.
Ultimately, this legal standoff marks a necessary shift in how we hold big tech accountable for the real-world consequences of the software they gatekeep. It is no longer enough for these companies to publish “developer policies” that ban pornographic content while pocketing the revenue from the very apps that violate those rules. Accountability requires a deeper level of oversight, proactive policing, and an uncompromising stance against services that facilitate abuse. As we move into an era where AI-driven content will become even more pervasive, the demand from San Francisco is a message to all tech leaders: the era of profiting from exploitation under the guise of neutral platform hosting must come to an immediate and permanent end.
