How an OnlyFans Model and a Cosplayer Are Fighting Nonconsensual Deepfake Porn

Episode Transcript

This is a computer-generated transcript. While our team has reviewed it, there may be errors.

Morgan Sung: Are you closing your tabs? You can be honest, this is a safe space. If you’re a fan of Close All Tabs and you want more of it, then please rate and review us on Spotify, Apple Podcasts, or wherever you listen to the show. And tell your friends about us. It would be such a huge help to get the word out.

Okay, let’s get to the show. Just a note, this episode contains mentions of gender-based violence and non-consensual intimate imagery, which may be triggering for some listeners.

So, you know Grok? It’s the AI chatbot integrated with X, the social media app formerly known as Twitter and now owned by Elon Musk. Well, since late last year, Grok has been embroiled in an undressing scandal, generating sexually explicit images of people without their consent. The majority of targets were women. Some were minors, young girls. For a few weeks, it was a pretty disgusting widespread trend. When women or even teenage girls posted fully clothed photos of themselves on X, other users would comment and tag Grok, asking it to ‘put her in a bikini’ or ‘take off her top.’ The chatbot would publicly respond with a generated lewd or completely naked image of the subject. Some users went even further, asking Grok to add blood and bruises, prompting the chatbot to generate graphic, sexually violent images of these women.

Morgpie: Oh man, it was very much like I was waking up every day and I didn’t want to post.

Morgan Sung: This is Morgpie, a Twitch streamer and OnlyFans creator. People who know her IRL call her Morgan. She’s been a porn actress for years, and as someone who makes sexually explicit content, she’s used to creeps harassing her with her own nudes. But the Grok and dressing trend really unsettled her. It was the worst in January.

Morgpie: Being looped in with something that is so violating, and like you said, something that’s even affecting minors is just disgusting. Every day I was going into my comments and just like hiding replies and blocking because I’m like, I’m not going to let you guys just generate these images of me that I did not consent to, especially if it’s being associated with basically creating child pornography on Twitter.

Morgan Sung: This was non-consensual, intimate imagery, more commonly known as deep fake porn. A deep fake is content that has been generated or manipulated by AI to imitate someone else. Zander Small, another content creator and a friend of Morgan’s, says that the proliferation of AI tools has started to seriously affect content creators, regardless of whether or not they make adult content.

Zander Small: Deep fakes can be anything from deep fake explicit imagery with like, a creator doing something or nude content that they didn’t consent to. Or it could be stuff as simple as like, an audio deep fake where a creator is saying something that they don’t consent too, which might have repercussions of them being canceled or stuff that they just obviously wouldn’t consent to saying.

Morgan Sung: Morgan hasn’t had to deal with deep fake porn of herself as much. After years of being in this industry, she’s developed thick skin. She’s mostly dealt with leaks, or explicit content that she posted behind a paywall that was illegally downloaded and posted elsewhere, without her consent. But the Grok trend is just the tip of the iceberg. Non-consensual deep fake-porn has exploded over the last few years.

Morgpie: I think that for a lot of people, the lack of consent is very attractive.

Morgan Sung: This is an issue that overwhelmingly affects women, and these circles are not as fringe as you might think. An annual report last year by the cybersecurity firm DeepStrike found that roughly 97% of all deepfakes online fall under non-consensual intimate imagery, and that 99 to 100% of victims of deepfake pornography are women. Here’s Zander again.

Zander Small: I think it is either fans, if you want to call them that, or just creeps on the internet, wanting to see more out of a creator than they consented to. I know it affects a lot of SFW creators.

Morgan Sung: SFW, or Safe for Work. They don’t show nudity or make sexually explicit content. While NSFW, not Safer work, means adult content.

Zander Small: Uh, you know, and I guess from that, you know, if a creator isn’t consenting to do more explicit content, then, you know, these, uh, I guess perpetrators, creeps, whatever you want to call them, you know, take into their hands to do it themselves. And it’s incredibly easy to deep fake content and, you know, as models get better and better and they get quicker and quicker, it doesn’t really require as much of sophisticated technology to run these models.

Morgan Sung: Some of the mainstream models, ChatGPT, Gemini, Claude, have guardrails that are supposed to prevent them from generating deep fake porn. In January, X announced that it implemented technological measures to prevent Grok from modifying images of real people in revealing clothing. But there are ways to get around these guardraills. Just last month, NBC News reported that Grok is still generating deep-fake porn of real women. And like Zander said, there are so many other models out there that just don’t have these guardrails in the first place.

Zander hasn’t had to deal with deep fake porn of himself, but he’s seen how much it’s affected people he’s close to, other safer work creators who don’t make explicit content. And Morgan, coming from the porn industry, has seen how this issue affects her fellow adult content creators.

So late last year, they teamed up to come up with a solution for other creators. Today, we’re diving into the seedy reality of non-consensual deepfake porn, when it got so bad, why it’s so hard to stop, and how two Gen Z content creators are trying to tackle it. Ready?

This is Close All Tabs. I’m Morgan Sung, tech journalist and your chronically online friend, here to open as many browser tabs as it takes to help you understand how the digital world affects our real lives. Let’s get into it.

Let’s open our first tab: the reality of non-consensual deep fake porn. Morgan is an award-winning porn creator. Literally, she has multiple Pornhub awards. And when she started years ago, the internet was very different.

Morgpie: It’s very interesting because when I first started, the climate was very much like, if you opened up Twitter, you would see tweets that are like, ‘sex work is real work.’ Of course, this was kind of around the time when OnlyFans was only just emerging barely.

Morgan Sung: In the world of adult content, there was before OnlyFans, and then there’s after OnlyFans. The platform completely changed the game, lowering the barrier of entry for new creators and giving them new options to monetize their content. Morgan said that before OnlyFans blew up, the only way to make a living as an independent porn creator was to land on the front page of Pornhub, or actresses had to break into the industry by being part of studio productions where they didn’t have as much autonomy.

Morgpie: It’s very interesting the shift between whenever porn was basically widely available, you didn’t really have to pay much for it. When I first started, I was uploading to Pornhub, and that was full length, full scenes that you could see for free at any time. Whereas now, the climate has shifted a lot to where creators like myself have a lot more control. So we’re able to, you know, use OnlyFans as a platform where we are more connected with our audience and that is actually the main pull. Now we’re in this age where these models can kind of take a bit of that control back. They can control what content they make, how much they sell it for. And I think that that plays so much into like the conversation about deepfakes where it’s about control. It’s all about consent. And then with deepfakes, you can make anybody do anything. So you have the control over this other person.

Morgan Sung: Since joining OnlyFans, Morgan and other adult creators have dealt with the same problem: leaks. They consent to paying subscribers accessing certain premium content that’s been posted behind a paywall. Then some unscrupulous subscriber downloads it and posts it publicly without their permission for the rest of the world to see. It was a constant source of frustration for Morgan.

And then about a year and a half ago, Morgan noticed the deep fakes. Her friends told her about how they stumbled across videos of themselves online, but it wasn’t really them. Someone had taken explicit content from behind their paywalls and modified it, morphing them into these scenarios that the creators never wanted to be in.

Morgpie: Again, it all stems back to control. It’s like, ‘oh, you did this thing that I didn’t like. Well, look at this control I have over your image. I’m going to use that against you.’.

Morgan Sung: I think some detractors would say, like, ‘oh, well, if you make explicit content, why does deep fake porn bother you? Or why do your leaks bother you?’ What would you say to them?

Morgpie: I mean, it’s it’s all about consent. That’s like saying, ‘oh, because you make porn, if I see you on the street, I can sexually assault you.’ You know, it’s like, consent is a very real thing. And there’s a big difference between me in the comfort of my own home within my own boundaries, producing content that I enjoy, and somebody else taking these things and making content that I didn’t consent to be in.

Morgan Sung: It’s not just Morgan and her fellow porn actresses dealing with this. Women who don’t make explicit content are also subjected to this harassment. One of the most well-known cases of this was when Atrioc, a Twitch streamer, was live. During his stream, he showed his open tabs for a split second, and one of them included deep fake porn of his own friends and streaming colleagues. He was caught buying this content. QTCinderella, another streamer was one of Atriok’s close friends. She was also a victim of the deep fake porn he bought.

[Audio Clip of QTCinderella]
Atrioc for showing it to thousands of people, the people DMing me pictures of myself from that website, f*ck you all!

Morgpie: Pokimane is like a great example of this.

Morgan Sung: Pokimane is another Twitch streamer who was also a victim of atriox deep fake porn purchases. She does not make explicit adult content, but as a woman existing online, she deals with harassment constantly. Like, here she is reading comments from her audience.

[Audio clip of Pokimane]

Yo yo yo, let’s see some ass. This ain’t a club fam, this is just my Twitch chat.

Morgpie: There are so many clips of her literally just getting up and  standing up out of her chair and that’ll get clipped and posted all over Twitter. And all of Twitter is like, ‘look at what she’s doing. She’s gooner baiting!’.

Morgan Sung: Goonerbait started as a term to describe video games or anime that aren’t pornographic but contain a lot of sexual imagery like jiggle physics and very scantily clad female characters. It’s media designed to appeal to gooners. Gooners are porn addicts. And now, internet randos love to accuse real-life women of gooner baiting, mainly female streamers like Pokimane.

Morgpie: ‘She’s, you know, performing for her male audience.’ And it’s like, well, is she really doing anything? She kind of just got up and walked out of the room, but they’re like, ‘oh, her pants are a little too tight.’ So it’s, like, I think this idea of a woman that’s kind of, just not really even doing anything, a lot of people love to just over-sexualize.

Morgan Sung: In some online circles, there is the sentiment that women like Morgan deserve to be deepfaked because they already make porn, and that women, like Pokimane, also deserve to deepfake because they’re somehow gooner baiting. It even affects people who don’t post online. Non-consensual deepfakes are rampant in schools. A Wired investigation last month found that high school boys have targeted their fellow classmates by spreading fake, generated nudes of them. These are teenage girls.

Morgpie: Yeah, the thing is, it’s not going to stop with sex workers. As much as the sentiment these days is very anti-sex work, like, ‘oh, if you make this content, you’re kind of putting yourself up to be distributed in this way.’ But the thing it is, is it’s 100% a slippery slope and it’s going to keep going into Twitch streamers who are known and even just normal people. There’s nothing stopping anybody from pulling up somebody’s Facebook profile, just a normal person who doesn’t produce any content whatsoever, and making explicit deep fakes of them and distributing them. And that can be used as blackmail. The possibilities there are quite literally endless in terms of the harm that they could cause for everybody.

Morgan Sung: You’ve talked about spending so much money on deepfake takedowns, but how did you initially try to tackle this problem of deepfakes and leaked content?

Morgpie: I was going in every single week and I was Googling my name and I was going on like Twitter, Reddit, all these other sites, just like searching for my name, um, and seeing pages and pages and pages of all this leaked content that would come up. And back then I was paying over a thousand dollars a month on these takedowns, but I would still have to go in and manually report a lot of stuff. You shouldn’t really have to go in and look at your own leaks and your own deep fakes, which is just awful.

Morgan Sung: Morgan was at her wits end. And then, late last year, she saw that Zander was working on a project that may be able to solve her problem. And she wanted to help. We’ll hear Zander’s story after the break.

But first, we wanted to remind you that Close All Tabs depends on listeners like you to keep us going. You can support us by becoming a member at donate.kqed.org slash podcasts. Okay, back to the story after the break. Stick around.

We’re back. Now, let’s open that new tab: What is Fanlock?

Zander had started out as a Minecraft YouTuber back in high school. It was a fun thing he did on the side before he went to college to study software engineering. He was on his high school robotics team and loved tinkering and fixing things. A few years ago, during his sophomore year, he started going to anime conventions with his friends. Here’s the thing, Zander’s really tall. He’s 6’8″. His friend pointed out that he could carve out a real niche as a comically tall cosplayer. He pushed Zander to start posting.

Zander Small: He was like ‘Bro, it’s gonna be like viral because like, oh my gosh, why is a Gojo cosplayer like as tall as like LeBron James?’ So I did it and it did pretty good. And I guess it just snowballed from there and I just haven’t stopped since.

Morgan Sung: About six months after he went viral as comically tall Gojo, he started getting brand deals from anime companies. He gained hundreds of thousands of followers. He flew all over the country, attending cons and meetups. He even hosted a few lookalike competitions. There’s a picture of the Hatsune Miku lookalite competition he hosted. A gaggle of cosplayers in turquoise wigs, and then Zander, towering above the crowd in his own turquois getup. Of course, he was still in school juggling a burgeoning full-time career as a content creator while also attending classes and doing homework and studying for exams.

He considered dropping out, but his parents really, really wanted him to stay in school. They weren’t thrilled at the idea of their son leaving an engineering degree to pursue anime content. So he stuck it out, and last year, while finishing up his last semester of school, He stumbled across this deep fake problem. It struck a very personal chord.

Zander Small: So about a month before I graduated, my girlfriend, who’s an SFW creator, had a huge deep fake problem. Um, you know, there’s accounts popping up on like Threads or Instagram that either use her likeness or just full on non-consensual porn, uh, deep fakes of her, which is super mentally taxing, uh on her, you know, as an SF W creator. You know, she didn’t consent to being in those positions or having these account to DM her fans, like, ‘Hey, send me $400 and we’ll go on a date,’ type of just scam content. So it was from there that I was like, let me see what’s up and see if I can help you. So that’s when I  really took a deep dive into DMCA, non-consensual imagery and depending on the platform it’s on what you can do about that.

Morgan Sung: DMCA, as in the Digital Millennium Copyright Act. It’s copyright law for internet content.

Zander Small: And I was able to get a lot of her stuff down, which was great. Uh, but then at that point it was like, you know, what are the other players in the space doing about this?

Morgan Sung: What he found were takedown tools that were very expensive and not that effective. While creator management firms and talent agencies have in-house services for this, they’re inaccessible to smaller creators. After Zander helped his girlfriend, her friends reached out to him. They had the same problem. And then their friends reached out. And all of this coincided with his post-graduation job search. He planned to at least try to use his degree. But the job market for entry-level software engineers was rough.

Zander Small: I think by the third final round interview at like some fang company where they rejected me after four weeks and five interviews, I was just so fed up. I was like, you know what, screw this. I’m gonna just do this myself. I’ma make my own company. So, and at that time, you it’s like the overlap of like, oh, I figured out how to do this. I could help more creators like this and really solve a real problem.

Morgan Sung: And so, he started working on it, a tool for creators that would scan the internet for leaked and deepfaked content and automatically send DMCA takedown requests. And if the sites didn’t comply, this tool would have to find other ways to force a takedow.

Zander knew how traumatic it was for his girlfriend and her friends to be constantly confronted with non-consensual deepfake porn. So, he wanted this tool to take down content automatically, without creators having to see it. And the tool also had to catch the non-consensual deepfakes before they spread to other platforms. But he knew he couldn’t do it alone. He needed the perspective of other creators for it to really work.

Zander Small: I pretty much just posted on my close friends at some point, like, hey, I’m thinking about doing this as like an actual like business or something like that. If anyone will be down to just test it out for free and see how good like my, you know, scanning architecture and stuff like that is, let me know. And Morgan actually swiped up on the story and was like, hey, that actually sounds pretty neat. I’d be down.

Morgan Sung: Morgan and Zander had met at TwitchCon a while back.

Zander Small: And we hopped on a call and I was like, ‘would you be down to like do this with me?’ Cause like, I think it’d be pretty sick if we had like two creators doing it that know the problem. You know, Morgan knows firsthand, like the adult space, but as well as like a firsthand account of like leaks and deep fakes and you know, where they live and stuff like that. And you know I guess from there, it just was one of those things where it was like I think this could be a real player in the space and I’m really passionate about it.

Morgan Sung: Morgan, what was it like for you to see that story?

Morgpie: I’ve struggled with this stuff for so long. I know so many people that I could tap in on and get their feedback. My scope in this space is so wide because I’ve had my eggs in so many baskets online And that I knew that I would be able to bring a good perspective and good input.

Morgan Sung: So they managed to raise $200,000, and with that, Morgan and Zander launched Fanlock earlier this year. Zander handles the technical side, making sure Fanlock works, and Morgan handles the creator side, managing outreach to other creators.

Zander Small: I guess I get to apply that degree that I was considering dropping out to do content for. And it’s, I guess like a full 360, you know, everyone that was like, you should stay in school and finish it out. I guess it came back to be useful because now I can apply it to helping my friends and other people in the space with this really real problem that they have.

Morgan Sung: This solution isn’t that straightforward though. That’s a new tab: Why is it so hard to take down deepfakes?

In May last year, President Trump signed the Take It Down Act, a landmark law that criminalizes the publication of non-consensual intimate imagery, including digital forgeries, aka deep fake porn. It’s one of Congress’s first bipartisan actions to tackle AI-generated content. The law also requires online platforms to implement a removal request system and to take down deep fake porn within 48 hours of a request.

Zander Small: A lot of these sites thankfully already had like forms or different reporting mechanisms to report deepfakes, but I think with this act itself, it’s a really good step in the right direction to combat non-consensual deepfake and, you know, props to the government for doing something right for once and actually passing this really quickly.

Morgan Sung: So the Take It Down Act is only enforceable under U.S. jurisdiction, although the EU also has similar laws. But a lot of these sites are based outside of these places, like in Russia.

Zander Small: Yeah, so for like Russia and Chinese sites, it gets a lot harder because they don’t have any need to comply either like deep fake penalties or DMCA because it’s specifically like USA, EU jurisdiction typically. And that makes it a lot hard to get content down off those sites if it’s even possible at all.

There’s a few things you can do for these sites. There’s been some sites I know firsthand that they use, let’s say, a USA-based company for their notification system. We’re able to submit basically a DMCA to those companies, basically being like, hey, just so you know, you’re aiding in copyright infringement by working with this client. If we were to take it a step further, we could always issue a DMC subpoena to them if they use Google Analytics, for example, to straight to Google. And that would help us get more information about… The actual emails of the site, who this person actually is. So if they’re in the EU or USA, we can take those legal routes. Obviously there’s sites I know that are pretty much, they’re built from the ground up for piracy and it’s pretty much impossible to get those stuff down.

Morgan Sung: One of the more difficult aspects of tackling deepfakes is catching them before Google indexes them, basically, storing web pages in its own database so they appear in search results. Because when something appears in search results, it spreads like wildfire.

Google updated its search functions a few years ago to identify deepfakes and prevent them from appearing at the top of search results, but there are still deepfakess that slip through the cracks. Zander said that Fanlock keeps tabs on specific sites that have histories of hosting non-consensual deepfakes. They scan them and send takedown demands, before they hit Google search results.

Zander Small: You know, no one wants their family Googling them or something and they see deep fakes of them all over Google Images.

Morgan Sung: I know Fanlock also relies on a lot of facial recognition technology to identify leaked content and deepfakes. Obviously, this technology is very controversial. It’s often used in law enforcement and has a lot connections to surveillance. But what are your thoughts on this use of facial-recognition technology?

Zander Small: Yeah, I mean, obviously, if a creator signs up for our platform and we’re doing it in a consensual manner, I think that’s great. I obviously am big anti-surveillance, but I think the the key word at the end of the day is just consent, which is like the fundamental problem that I think these creators are having. And if they’re consenting to a service to take down stuff that was made non-consensually, I think, that’s why our creators are okay with it. And I think there’s a big differentiation between that and then, you know, some tech company scanning my face to see if I’m a criminal or something like that.

Morgan Sung: I want to talk about some of the technical challenges that still exist. You mentioned trying to build a Telegram scanner right now. A lot of non-consensual deep fake porn is passed around in closed channels on Discord or group chats or Telegram. Do either of you have any experience with this happening? Like, what is the approach here?

Zander Small: When we were building FanLock, I was like, Telegram is, like, the final boss of piracy. I really want to build a solution that while we can’t scan a hundred percent of Telegram, I want to build the absolute most, like I guess comprehensive Telegram scanner we can based on like what’s publicly available and what providers there are to us. So for Telegram, typically for like private groups and stuff like that, you’re able to join them if you have like a join link, which we’ve kind of gotten from people being like, ‘hey, I got leaks here, join my channel.’ And after we get the join link we’re able to figure out where copyrighted content is.

We already do have our Telegram scanner up. You know, we have about 11 million channels, you know, from our own services, but also third party providers that we use that have kind of indexed Telegram for us, which is great. Discord is a little bit trickier because it’s a TOS breach to use any sort of like bot activity on that.

 Morgan Sung: TOS is Terms of Service, the contract between a platform like Telegram and its users.

Zander Small: For now, like on Discord, if someone has a link that they’ve noticed that they want down, they can submit it to us and then we can do it from there. We currently don’t scan Discord because it is like a TOS breach to do, but we’re hoping as, like I said, as we grow that door can open.

Morgan Sung: Overall, what hurdles still exist when it comes to taking down deepfakes? Like what’s the kind of like technical white whale you’re still chasing?

Zander Small: Yeah, I’d say the biggest thing that we’re trying to roll out is actually identifying who leaked or who deep faked XYZ content. I think if we were able to do that, we might, I wouldn’t say solve the piracy problem, but definitely lower it. You know, we’re really hoping we can get in talks with, you know, platforms like OF, Fansly or Instagram and stuff like that, uh,  to roll out a technology that we’re working on where basically it embeds like an invisible watermark into different images and stuff like that.

So if it is leaked or if it has deep faked or if someone else’s face has put on it, they’re able to know who exactly posted it based off this invisible embedded technology, which already exists for sites like Netflix. It’s how they track like video, uh, I guess leaks or, you know, from studios that maybe have like a trailer for the new Avengers movie and they want to track if it got leaked on X or anything like that.

I think if we’re able to get that done, like I feel like we’d significantly fix the problem and be a lot more proactive. Because I mean, if people start realizing, ‘oh shoot, if I leak or deep fake content, my account gets banned. Like, it’s going to really throw a wrench in the whole leak ecosystem. And that’s what we’re really trying to build towards right now.

Morgan Sung: You’re coming from very different sides of the internet, kind of, whether in the safe work side or the adult content industry. But this is also a problem that deeply affects both of your spheres of the creator economy. How has the proliferation of deepfake porn changed the creator industry for you? And what would you say to someone who’s afraid to keep posting?

Morgpie: The unfortunate thing is it’s such an uphill battle when it comes to deepfaked and leaked content, especially with AI getting as good as it is right now. But to somebody who is kind of scared to post right now, just know that there are people who are trying to find solutions to this kind of stuff.

And for these people who are generating this kind of content, it’s very much about their own sense of control. It doesn’t reflect you as a creator. You shouldn’t be afraid to post what you want because of this horrible threat of somebody taking your content and basically twisting it into something that you didn’t consent to. And hopefully our government can kind of catch up with this kind of stuff here pretty soon. But there are people like me and Zander who are trying to take real steps to help mitigate this.

Zander Small: For creators, I’d say, you know, if you need to, you know, get anything you need for support on it, do it. You know, if you need to take a step back, do it. And then I’d say like, it’s a twofold thing where it’s like, don’t glamorize generative AI video and image content because that only speeds up the industry and then really push for better legislation and, you know, call your Senator, call your Congressman, like get it passed. Because It’s only going to get worse as it gets easier and it’s able to be done for more people. I think those are probably the two biggest things a creator can do right now that has like an actual like tangible impact to halt this problem or make it slow down at least.

Morgan Sung: Well, thank you both so much for talking about all of this.

Morgpie: Yeah, thank you for having us.

Zander Small: Yeah, for sure.

Morgan Sung: If you or someone you know has been targeted with deep fake porn, there are ways to have it removed. Fanlock also has free guides for creators navigating this problem. Check the show notes for more. We’ll link to a few resources about the Take It Down Act and how to remove non-consensual intimate imagery. For now, let’s close all these tabs.

Close All Tabs is a production of KQED Studios and is reported and hosted by me, Morgan Sung. This episode was produced by Chris Egusa, who also composed our theme song and credits music. It was edited by Chris Hambrick. The Close All tabs team also includes producer Maya Cueva and audio engineer, Brendan Willard. Additional music by APM. Audience engagement support from Maha Sanad. Jen Chien is our director of podcasts and Ethan Toven-Lindsey is our editor in chief.

Some members of the KQED podcast team are represented by the Screen Actors Guild, American Federation of Television and Radio Artists, San Francisco, Northern California local.

Keyboard sounds were recorded on my purple and pink dust silver K84 wired mechanical keyboard with Gateron red switches. Thanks for listening.