Unlock Editor’s Digest for free
Roula Khalaf, editor-in-chief of the FT, selects her favorite stories in this weekly newsletter.
Elon Musk’s Grok AI model lacked safeguards to prevent users from generating sexualized deepfakes of women and children, according to experts who warn that many AI systems are vulnerable to producing similar material.
Friday, the billionaire’s start-up xAI said it limited the use of its Grok image generator to paying subscribers only. The move follows threats of fines and bans from governments and regulators in the EU, UK and France.
The company, which acquired Musk’s social media site X last year, is an exception, designing its AI products to have fewer resources. “guardrail” content than competitors such as OpenAI and Google. Its owner called his Grok model “maximum truth-seeking.”
“The way the model was built and the lack, it seems, of security restrictions and alignments… means you’re inevitably going to have cases like these,” said Henry Ajder, an expert on AI and deepfakes.
xAI did not respond to a request for comment. Musk has previously stated that “anyone who uses Grok to create illegal content will face the same consequences as if they download illegal content.”
The news comes as AI companies, faced with the rising costs of developing the infrastructure needed for their ambitions, are under pressure to step up their commitment and monetize their products. Other groups are considering allowing more sexual content. OpenAI, for example, announced plans to launch an “adult mode” for its chatbot this quarter.
Although xAI has not shared details on how it trained its model, it is very likely that it was trained on a large dataset of images scraped from the internet.
In 2023, researchers at Stanford University discovered that a popular open source database, LAION-5B, used to create AI image generators, was full of child sexual abuse material (CSAM).
The dataset also contains pornographic content as well as violent, racist and sexist images. As these datasets contain billions of images, it is difficult for AI labs to remove or filter out any offensive content.
Experts added that even if xAI ensured its model was not trained on CSAM, it’s possible the model could still generate sexualized images of children through a technique called “style transfer.”
If a model is trained in depictions of nude people, it has the ability to transfer those images to a photo of a clothed adult or child.
AI companies have limited ways to prevent users from generating harmful content, such as adding a security filter on top of the model that blocks certain keywords.
These are often crude tools, which users can bypass by using written prompts with, for example, alternate spellings to “jailbreak” the template.
In 2024, Google came under fire after its Gemini image generation model created images of black Nazis, after users asked the system “German soldier in 1943” using a misspelled variation of the prompt.
Companies can also use AI tools to detect unwanted characteristics of images, such as nudity and blood, once they have been made and prevent users from accessing them.
AI companies can also remove certain “concepts” from the model or modify the models themselves to ensure that the model only generates non-harmful images.
However, these techniques are not perfect, often fail when used at scale, and are vulnerable to attackers.
Grok 4, xAI’s latest and most powerful model, was launched in July and has a “Spicy Mode” feature that allows users to generate sexually suggestive content for adults.
Another problem is that xAI has integrated some Grok features into the social network X, allowing more images to appear publicly and be widely distributed.
Grok also has a video generation model, capable of generating graphic and extreme content, but which is not available to users on X.
Since acquiring X, formerly known as Twitter, in 2022, Musk has sought to ease security and restrictions on the social network. Musk fired Twitter’s ethical AI team, which was working on techniques to prevent the spread of harmful content on the platform.
Charlotte Wilson, head of cybersecurity firm Check Point Software, said more technical controls needed to be put in place, including “stronger content classifiers, repeat offender detection, rapid removal pipelines and visible audit trails.”
X’s response to the growing public outcry to restrict image generation to paying users has only provoked further backlash.
Refuge, the UK’s largest domestic violence charity, said it represented a “monetisation of abuse” which “allowed X to profit from the harm”.
,
