Last summer, a pornographic video from a prominent creator of TikTok went viral on social media. A screenshot of the video on TikTok went to the Instagram account @tiktokroom, accompanied by a comment from the creator whose face was published in the video. “It’s a deepfake but still, my parents on this app, relax,” the creator said. (Rolling stone does not name this creator or any other creator referenced in this story, on the grounds that the videos were posted without their consent.)
A deepfake is a digitally manipulated video that uses artificial intelligence technology to swap a person's face (often, but not always, a celebrity) for another person's body, usually without their consent. Much of the media attention to deepfakes has focused on their potential for spreading disinformation, particularly in relation to a political campaign or election, as one Nancy Pelosi's doctored clip which went viral on Facebook last year. Yet the vast majority of deepfakes - 96%, according to 2019 research from artificial intelligence firm Sensity - are pornographic and used to target women.
In the past, deepfake technology was primarily used to create nude or sexually explicit images of female celebrities in particular. Part of that is because to create deepfake models you need thousands of photos or several minutes of video footage to do a good job, says Giorgio Patrini, Founder and CEO of Sensity.
But the technology has become more accessible and less complex, with terrifying results. Last week, Sensity published a report on an AI bot on encrypted messaging app Telegram used to create nude or sexually explicit images of hundreds of thousands of women without their consent. Unlike other software used to create deepfake videos, the bot only required the use of a photo to depict nude women, leading to the increasing targeting of non-celebrity women and girls . The same goes for influencers on platforms like Instagram and TikTok. “It’s a video-based platform, so it actually provides more material [to make deepfakes],” Patrini says. “These people are at greater risk than if they were just uploading photos. »
Danielle Citron, professor of law at Boston University Law School and vice president of the Cyber Civil Rights Initiative, says the psychological impact of appearing in a pornographic deepfake video cannot be overstated, and some have also compared non-consensual pornography to a form of digital rape. “When you see a deepfake sex video of yourself, it feels like it's your body viscerally. It is an appropriation of your gender identity without your permission. It sounds like a terrible violation of autonomy and the body, and it makes room in your head, ”she says. According to Patrini, appearing in a deepfake porn video can have a significant and deleterious financial impact on a creator's business model, citing the example of a prominent YouTuber who lost a partnership with a brand after someone posted a deepfake video of her on a porn site.
To make matters worse, almost a third of users of TikTok are under 14 years old, according to internal company data reviewed by the New York Times making this population even more vulnerable to the threat. As Rolling stone previously reported, it is not uncommon for minor creators of TikTok find their videos posted in compilation videos on tube sites like Pornhub. Rolling stone spoke with the mother of a 17-year-old girl who discovered that one of her videos TikTok had been published on Pornhub. “She was mortified. She didn’t want to go back to school,” the mother, who asked to remain anonymous, told me. “She posted this video very innocently. She didn't want to get involved with Pornhub. This is not a lesson you should have to learn at 17. (She says Pornhub immediately removed the video at their request. In an email to Rolling stone, Pornhub denied that the content of TikTok was frequently posted on its platform.)
In our investigation for this story, Rolling stone found more than two dozen examples of prominent creators of TikTok featured in deepfake porn. Most of them are initially posted on one of the few websites devoted exclusively to publishing deepfakes, but they are also not difficult to find on social media platforms like Twitter and Reddit, although both Platforms have policies prohibiting deepfakes.
In a statement, a Twitter spokesperson directed Rolling stone to its policy banning deepfakes, or any “intimate photos or videos of someone that were produced or distributed without their consent.” At the time of publication, he did not delete a tweet containing a deepfake pornographic video from two creators of TikTok. Reddit, which also has a policy against posting non-consensual deepfakes, did not immediately respond to request for comment.
They are also not hard to find on Pornhub and other tube sites, although Pornhub also prohibits deepfakes. "Tthey just banned people from labeling deepfakes. People could still download them and use a different name, ”says former Sensity researcher Henry Ajder. He reckons that "you probably have somewhere in the few thousand" deepfakes on Pornhub.
One of the most popular websites for deepfakes, which Rolling stone refuses to name, had a Discord server with over 100 members where users could ask their celebrities and influencers TikTok favorites to be the subject of a deepfake. Although this website only posts videos featuring creators over the age of 18, one of the most requested celebrities on this server is a popular creator of TikTok who just turned 17. [name redacted], by the way, she will be 18 in 4 days,” one user recently said, asking a creator. (The administrator made the video and posted it on the website two weeks after she turned 18.)
When asked about the server, Discord said it violated its policies regarding non-consensual nudity and removed it. “Discord has a zero tolerance approach to non-consensual pornography on the service and the company takes immediate action when we become aware of it,” a company spokesperson said. A notice on the deepfake pornography website, however, states that it "works on an alternate server."
As to why people create deepfake porn in the first place, Citron says some mistakenly believe it's essentially victimless crime. “Evil is so far removed from people that it feels like a game,” she said. “You can make someone's body your own and you don't have to ask them. The phenomenon of deepfake pornography stems from a "culture of impunity", or the idea that young men have the right to do whatever they want with women's bodies.
But while some men may feel empowered to do whatever they want with the influencers' images, being public figures, ultimately the subject of deepfake porn, Citron says, " like a sexual assault. "
