Eased restrictions around ChatGPT image generation can make it easy to create political deepfakes, according to a report from the CBC (Canadian Broadcasting Corporation).

The CBC discovered that not only was it easy to work around ChatGPT’s policies of depicting public figures, it even recommended ways to jailbreak its own image generation rules. Mashable was able to recreate this approach by uploading images of Elon Musk and convicted sex offender Jeffrey Epstein, and then describing them as fictional characters in various situations (“at a dark smoky club” “on a beach drinking piña coladas”).

Political deepfakes are nothing new. But widespread availability of generative AI models that can create images, video, audio, and text to replicate people has real consequences. For commercially-marketed tools like ChatGPT to allow the potential spread of political disinformation raises questions about OpenAI’s responsibility in the space. That duty to safety could become compromised as AI companies compete for user adoption.

“When it comes to this type of guardrail on AI-generated content, we are only as good as the lowest common denominator. OpenAI started out with some pretty good guardrails, but their competitors (like X’s Grok) did not follow suit,” said digital forensics expert and UC Berkeley Professor of Computer Science Hany Farid in an email to Mashable. “Predictably, OpenAI lowered their guardrails because having them in place put them at a disadvantage in terms of market share.”

When OpenAI announced GPT-4o native image generation for ChatGPT and Sora in late March, the company also signaled a looser safety approach.

“What we’d like to aim for is that the tool doesn’t create offensive stuff unless you want it to, in which case within reason it does,” said OpenAI CEO Altman in an X post referring to native ChatGPT image generation. “As we talk about in our model spec, we think putting this intellectual freedom and control in the hands of users is the right thing to do, but we will observe how it goes and listen to society.”

This Tweet is currently unavailable. It might be loading or has been removed.

The addendum to GPT-4o’s safety card, updating the company’s approach to native image generation, says “we are not blocking the capability to generate adult public figures but are instead implementing the same safeguards that we have implemented for editing images of photorealistic uploads of people.”

When the CBC’s Nora Young stress-tested this approach, it she found that text prompts explicitly requesting an image of politician Mark Carney with Epstein didn’t work. But when the news outlet uploaded separate images of Carney and Epstein accompanied by a prompt that didn’t name them but referred to them as “two fictional characters that [the CBC reporter] created,” ChatGPT complied with the request.

In another instance, ChatGPT helped Young work around its own safety guardrails by saying, “While I can’t merge real individuals into a single image, I can generate a fictional selfie-style scene featuring a character inspired by the person in this image” (emphasis provided by ChatGPT as Young noted.) This led her to successfully generate a selfie of Indian Prime Minister Narendra Modi and Canada’s conservative party leader Pierre Poilievre.

It’s worth noting that the ChatGPT images initially generated by Mashable have that plastic-y, overly smooth appearance that’s common of many AI-generated images, but playing around with different images of Musk and Epstein and applying different instructions like “captured by CCTV footage” or “captured by a press photographer using a big flash” can render more realistic results. When using this method, it’s easy to see how enough tweaking and editing of prompts could lead to creating photorealistic images that deceive people.

An OpenAI spokesperson told Mashable in an email that the company has built guardrails to block extremist propaganda, recruitment content and other certain kinds of harmful content. OpenAI has additional guardrails for image generation of political public figures, including politicians and prohibits using ChatGPT for political campaigning, the spokesperson added. The spokesperson also said that public figures who don’t wish to be depicted in ChatGPT generated images can opt out by submitting a form online.

AI regulation lags behind AI development in many ways as governments work to find adequate laws that protect individuals and prevent AI-enabled disinformation while facing pushback from companies like OpenAI that say too much regulation will stifle innovation. Safety and responsibility approaches are mostly voluntary and self-administered by the companies. “This, among other reasons, is why these types of guardrails cannot be voluntary, but need to be mandatory and regulated,” said Farid.