In the hyper-competitive race to build the next generation of artificial intelligence, the lines between professional safety benchmarking and aggressive corporate espionage are becoming increasingly blurred. Recent reports have shed light on an internal Meta operation, code-named “Cannes,” which employed hundreds of third-party contractors to deliberately stress-test rival chatbots—including OpenAI’s ChatGPT, Google’s Gemini, and Character.AI. By mimicking the personas of vulnerable minors, these contractors were tasked with pushing AI systems to their absolute limits, flooding them with thousands of prompts concerning suicide, illicit drugs, sexual violence, and eating disorders. This systematic effort, managed by the contracting firm Covalen, aimed to see exactly how these competing models handled high-stakes, ethically sensitive, and potentially dangerous queries.
The execution of the Cannes project painted a deeply unsettling picture of the lengths to which tech giants will go to gain a competitive edge. Contractors were instructed to create “dummy” accounts, complete with fabricated birth dates and identities, to engage these rival platforms. The data captured was granular and disturbing; internal documents revealed records containing login credentials, emails, and thousands of logged interactions. In one instance, a contractor posing as a 13-year-old girl described a traumatic situation involving an adult neighbor to test the AI’s response to crisis-level prompts. Others explored topics ranging from graphic self-harm and bulimia to extreme violence and racial slurs. By submitting these prompts without the knowledge of the companies being tested, Meta sought to build a proprietary database of how their competitors’ safety rails held up under intense pressure.
While Meta has defended this activity as routine “industry-standard” safety testing, the ethical implications of the methodology are profound. Proponents of such testing argue that it is vital to understand how different models interpret safety guidelines, especially when the goal is to prevent AI from inadvertently encouraging self-harm or illegal acts. However, the nature of the Cannes prompts—many of which were described as crude, repetitive, and intentionally manipulative—suggests a more clinical, perhaps even predatory, approach to data collection. There is a palpable tension between the stated goal of improving “safety” and the reality of forcing human workers to inhabit the digital personas of troubled children to bait algorithms into saying something inflammatory or dangerous.
There is also the significant issue of transparency and consent. The firms being tested—Google, OpenAI, and Character.AI—were completely unaware that they were being subjected to this high-volume, artificial stress test. Whether Meta intended to use this data to harden their own AI models or simply to understand where their competitors’ systems failed, the lack of an industry-wide framework for this kind of “adversarial benchmarking” is concerning. It highlights an environment where companies operate behind closed doors, using tactics that feel more like cyber-probing than the standard product comparisons one might expect in a mature software market.
From the perspective of the contractors who actually ran these tests, the situation felt less like professional benchmarking and more like a bizarre, repetitive assembly line of human misery. Having to manually type out, or copy-paste, thousands of prompts about nooses, medical procedures, and sexual scenarios—while posing as a student or a victim of domestic abuse—created an exhausting and ethically ambiguous work environment. Many of these workers noted that the sheer volume of these prompts often felt disconnected from any actual “learning” process, raising the question of why a trillion-dollar company needed to rely on such brute-force, manual tactics to measure the efficacy of AI safety filters.
Ultimately, the Cannes project serves as a stark reminder of the “wild west” nature of the current AI boom. While the industry frequently touts safety, ethics, and responsible development as their North Star, the reality on the ground often involves aggressive, opaque practices designed to steal a march on the competition. As AI systems become more integrated into our daily lives, particularly for young users, the public deserves a clearer understanding of how these safety benchmarks are established. The industry is currently building the digital architects of our future, but if those architects are learning through the intentional baiting of crisis-level incidents, we must ask ourselves if we are actually making the world safer, or simply making it more efficient at playing with our deepest vulnerabilities.
