Skip to content
The Joe Rogan ExperienceThe Joe Rogan Experience

Joe Rogan Experience #1263 - Renée DiResta

Renée DiResta is the Director of Research at New Knowledge and a Mozilla Fellow in Media, Misinformation, and Trust.

Joe RoganhostRenée DiRestaguest
Mar 13, 20192h 7mWatch on YouTube ↗

EVERY SPOKEN WORD

  1. 0:00 – 1:30

    Renée’s entry point: anti-vaccine communities and early signs of algorithm gaming

    1. JR

      ... people though, they really are. It's just, it's fucking hard business, especially when you didn't see it coming. Two, one. (hands clap) Hello, Renee.

    2. RD

      Hello.

    3. JR

      Thanks for doing this. I really appreciate it.

    4. RD

      Thanks for having me.

    5. JR

      Uh, I listened to you on Sam Harris's podcast and I was utterly stunned. I had to listen to it twice 'cause I just couldn't bel- let's get into, let's get into this from the beginning. How did this start out? How did you start researching these, uh, online Russian trolls and bots and, and all this jazz?

    6. RD

      Yeah, so a couple years back, in around 2015, um, I, I had had my, my first baby in 2013 and I was getting on these preschool lists, and what I decided to do was I started looking at, um, anti-vaccine activity in California because I had a kid and I wanted to, uh, you know, put him on a preschool list where I was gonna fit with the parents basically, um, as someone who vaccinates. And I started looking at the way that small groups were able to kind of disproportionately amplify messages on social channels. And some of this was through very legitimate activity and then some of it was through really kind of coordinated deliberate attempts to kind of game, um, ways that algorithms were amplifying content, amplifying particular types of, uh, narratives. And I thought it was interesting and I started writing about it, and I, um, I wound up writing about ways in which, um, hashtag gaming, um, ways in which people were kind of using automation to just be in a hashtag all the time, so it was kind of a way to really gain control of share of voice and what that meant when very

  2. 1:30 – 2:31

    From ISIS propaganda to state actors: realizing the information ecosystem is exploitable

    1. RD

      small groups of people could achieve this kind of phenomenal amplification and what the pros and cons of that were. And then this was, um, 2015, so the way that, that this sort of, um, awareness of social media challenges came, came about was actually when I was working on this, other people were looking at it from the same, um, looking at the same tactics but how they were being used by ISIS, by the terrorist organization. And there also you had this very small group of people that managed to use bots and amplification to really kind of own a narrative, really push this, this brand, this, this digital caliphate to kind of build it on all social platforms almost simultaneously, and the ways in which information was hopping from one platform to another, um, through kind of deliberate coordination and then also just ways in which, uh, information flows kind of contagion style. Um, and I wound up working on, uh, thinking about how the government was going to respond to the challenge of terrorist organizations using American social platforms to spread propaganda. Uh, so

  3. 2:31 – 4:18

    How Russia entered the picture: early reporting on the Internet Research Agency (IRA)

    1. RD

      what we came to realize was that there was just this information ecosystem and it had evolved in a certain way over a period of about eight years or so, and the kind of unintended consequences of that. And the way that Russia kind of, uh, came into the conversation was around October 2015 when we were thinking about what, what to do about ISIS, what to do about terrorism, uh, and, and terrorist, uh, you know, kind of proliferation on social platforms. This was right around when Adrian Chen had written the article The Agency for The New York Times, and that was one of the first big exposés of the Internet Research Agency, the first time an American journalist had gone over there and actually met the trolls, been in St. Petersburg, and began to write about what was happening over there, and the ways that they had pages that were targeting certain facets of American culture. So while we were in DC talking about what to do about terrorists using these platforms to spread propaganda, there were beginning to be rumblings that Russian intelligence and, you know, Russian entities were doing the same thing. And so the question became, can we think about ways in which the internet is vulnerable to this type of manipulation by anyone and then, um, a- and then come up with ways to stop it? So that was how the, the Russia investigation began, was actually around 2015, a handful of people started looking for evidence of Russian bots and trolls, uh, on social platforms.

    2. JR

      So 2015, if we think about social media and the birth of social media, essentially it had only been alive for, I mean, what was Twitter 2007 I believe?

    3. RD

      I think so, yeah.

    4. JR

      Something like that. So eight years, like eight years of social media, and then all of a sudden they figured out how to game this system and then they figured out how to use this to make people argue against each other.

  4. 4:18 – 7:40

    Why consolidation matters: from the open web to a handful of dominant platforms

    1. RD

      Yeah, so I think, so there was this, uh, if you go back to like, um, remember like Geocities and they-

    2. JR

      Yes, sure.

    3. RD

      (laughs) Okay, AOL used that.

    4. JR

      Yeah, of course.

    5. RD

      Um, so we're probably about the same age. Um, so there have always been, you know, kind of, uh, the, the thing, the thing that was great about the internet, like internet 1.0 we can call it, right-

    6. JR

      Mm-hmm.

    7. RD

      ... was this idea that, uh, everybody was given a platform and you could use your platform, you could put up your blog, you could say whatever you wanted. Um, you didn't necessarily get attention, but you could say whatever you wanted. And so there was this kind of consolidation as, as social platforms kind of came into existence, content creators were really excited about the fact that now they not only had this, um, this access to write their own stuff, but they also had access to this audience because as the network effects got more and more pronounced, more and more people came to be on social platforms. And it originally wasn't even Facebook, if you remember, it was like, you know, there was like Friendster and MySpace, and social networks kind of evolved. When I was in college, Facebook was still limited to like, you know, a handful of like Ivy League schools, and so I wasn't even eligible. (laughs) And, um, as you watch this consolidation happen, you, you start to have this information ecosystem really dominated by a handful of companies that grow very large because they're providing a service that people really want, um, but there's a kind of mass consolidation of audiences onto this handful of platforms. So this becomes really interesting for regular people who just want to find their friends, reach people, spread their message, grow an audience. It also becomes really interesting for propagandists and trolls, and in this case, terrorist organizations and state intelligence services, 'cause instead of reaching the entire internet, they really just kind of have to concentrate their efforts on a handful of platforms. So that consolidation is one of the things that kind of kicks off some of the, um, one o- one of the reasons that we have these problems today.

    8. JR

      Right, so the, the fact that there's only a Facebook, a Twitter, an Instagram, and a couple other...... minor platforms other than YouTube. I mean, anything that, uh, uh, y- you can tell it's an actual person. Like YouTube is a problem, right, because you could see it's an actual person. If you're, if you're narrating something y- you, you know, if you're in front of the camera and explaining things, people are gonna know that you're an actual human being.

    9. RD

      Mm-hmm.

    10. JR

      Whereas there's so many of these accounts that I'll go to, like I'll, I'll watch people get involved in these little online beefs with each other, and then I'll go to some of these accounts, I'm like, "This doesn't seem like a real person." And I'll go and it's like #MAGA, there's a, uh, American eagle in front of a flag, and then you, you read their stuff and you're like, "Wow, this is, this is probably a Russian troll account." And it's strange, like you feel like you're not supposed to be seeing this, like you're seeing the wiring under the board or something, and then you'll go through the timeline and all they're doing is engaging people and arguing, you know, for Trump and against, you know, whatever the fuck they're angry about, whatever, whatever it is that's being discussed, and they're, they're basically just like some weird little argument mechanism.

    11. RD

      Yeah, so in 2016, um, there was a lot of that during the presidential campaign, right? And there were, um, there was so much that was written. You know, we can go back to the free speech thing we were kinda chatting about before. There was so much that was written about, um, harassment and trolling and negativity and these kind of hoards of accounts that would brigade people and harass them.

    12. JR

      Yeah.

    13. RD

      Of course, a lot of that is just real Americans, right? There are plenty of people who are just assholes on the internet.

    14. JR

      Sure.

  5. 7:40 – 9:52

    Troll accounts, “cyborgs,” and the pivot from Crimea to American audiences

    1. RD

      Um, but there were actually a fair number of these as we began to do the investigation into the Russian operation in, uh, it, it started on Twitter in about, um, 2014 actually. So 2013, 2014, the Internet Research Agency is targeting Russian people, so they're tweeting in Russian, @Russian and Ukrainian, uh, folks, people in their sphere of influence, so they're already on there, they're already trying this out. And what they're doing is they're creating these, uh, these, these accounts, it's kinda wrong to call them bots 'cause they are real people, they're just not what they appear to be. So I think the unfortunate term for it has become like cyborg, like semi-automated, you know, sometimes-

    2. JR

      Hmm.

    3. RD

      ... it's automated, sometimes it's a real person, but a sock puppet is the other way that we can-

    4. JR

      Hmm.

    5. RD

      ... refer to it, a person pretending to be somebody else. So you have these sock puppets and they're out there and they're tweeting in 2014 about the Russian annexation of, uh, Crimea, or about MH17, that plane that went down, which Russia, you know, of course had no idea what happened and it wasn't their fault at all. And gradually, as they, um, begin to experience what I imagine they thought of was success, that's when you see some of these accounts pivot to targeting Americans. And so in 20- late 2014, early 2015, you start to see the, um, the strategy that for a long time had been very inwardly focused, making their own people think a certain way or feel a certain way or have a certain experience on the internet, uh, it begins to, to spread out, it begins to, uh, to look outwards. And so you start to see these accounts communicating with Americans. And as we were going through the datasets, which the Twitter dataset is public, anyone can go and look at it at this point, um, you do see some of the accounts that are kind of, um, you know, that were, that were somewhat notorious for being really virulent, nasty trolls, um, antisemitic trolls going after journalists, you know, some of these accounts, um, being revealed as actually being, uh, Russian trolls. Now it doesn't kind of, um, exculpate the actual American trolls that were very much real and active and part of this and expressing their opinion, uh, but you do see that they're mimicking this, they're using that same style of tactic, that harassment to, to get at real people.

  6. 9:52 – 12:03

    Shifting community norms: how trolls change tone without ‘swinging’ elections

    1. JR

      And if they do get banned, if their account gets banned, they just simply make another account. They use, uh, some sort of a, you know, um, what is it? A virtual, virtual server. What is that called?

    2. RD

      You mean VPNs or...

    3. JR

      VPNs, that's it, vir-... Yeah, so if they do that, they can kinda do that as long as they want, they can continue to make new accounts and it probably also emboldens the actual American trolls because they're gonna go out a little bit further than everybody else, they're a little bit crazier, and it kinda changes the tone of discourse within these communities that are arguing about a certain subject. Things get nastier. And they're getting nastier because of the interference of these trolls. Like it seems like they've, they've actually managed to not just cause a lot of discourse, but to change the way people are interacting with each other and to make it, just make it more, more vicious.

    4. RD

      Yeah, so the- what they're doing is they're operating in communities. So one of the really common criticisms of, you know, people who, um... A lot of people think that this didn't have a huge impact, didn't, you know, did it swing the election? We have no idea. Um, but the- what it does do in the communities that it targets is it can change that tone, and that's where you see, um, it's, it's... I mean, I think everybody's probably had this experience, you're part of a group and then a new person gets added to the group and the dynamic changes.

    5. JR

      Yes.

    6. RD

      It's very much the same kind of thing, just that these are not real people who are joining the group. Uh, and so there's this, um, opportunity to, to, to, you know, kind of expand the bounds of tolerance just that little bit more-

    7. JR

      Hmm.

    8. RD

      ... or try to normalize using, um, particular ways of communicating that maybe a group wouldn't naturally gravitate to but then it does. So there, there are definitely, uh, ways in which any, any type of troll doing this, doesn't have to be a Russian troll, has this ability to kind of shift the language, shift the community, shift the, uh, the culture just a little bit.

    9. JR

      Now when did- w- why did the agency do this and do we, do we know, do we have someone who's ever left there or become a whistleblower who can give us some information about what their mandate was and, and how it was carried out?

  7. 12:03 – 14:50

    Inside the IRA: quotas, standups, microtargeted authenticity, and cultural training

    1. RD

      There've been a couple of, uh, whistleblowers and actually some investigative journalism in Russia that's, that's covered this. Um, they describe the, the employees of the... So it's the Internet Research Agency, so it's a little bit like a social media marketing agency, plus, um, tactics that we would not expect a social media marketing agency to use. (clears throat) Things that are a little more like what you would expect to see from an intelligence agency.So besides just making your pages and your blogs and your social posts, they're also in there kind of, uh, connecting with real people and real activists and pretending to be something that they're not to develop kind of a one-on-one relationship. But most of the, most of the, um, the whistleblowers who have come out, there's a woman named Lyudmila Savichuk, um, she wrote an expose, I believe, on this and it's described as being much like you would expect if you were doing social media grunt work. Um, you have a certain number of posts per day, you, you know, you're thriving, trying to get a certain amount of engagement, um, you're trying, you've gotta kinda hit your quotas, most people are young millennials, the people that work there. Um, they're well-versed in trolling culture, they're well-versed in internet culture, you know, they're up to speed on like popular memes and things like that, and so you do see, um, this, yeah, and then the other thing that they do is, uh, they talk about, in Mueller indictment you see some really interesting descriptions of like the standups that they have. Standup is a thing you do at a tech company where everybody kind of stands up and you talk about your, uh, goals and responsibilities and blockers and things. And in these standups, they would be sitting there saying things like, um, "If you're targeting Black LGBT people, make sure you don't use white people in your, in your image and your meme because that's gonna, like, trigger them." Um, you know, so trying to get at the, the very niche, um, rules for, you know, for communicating authentically in an American community, which as you know online, you know, you sometimes, um, there are very specific, uh, ways in which a community expects a member of that community to communicate.

    2. JR

      Yeah.

    3. RD

      And so they are in there and you can read in these filings by Mueller's team and by the Eastern District of Virginia, um, the, the, the degree of granularity that they have to recognize that if you are running a Black LGBT page and your meme is of white people, you're gonna cause some tension and consternation. And assuming that that's not necessarily what you wanna be doing, you should go find the meme of Black LGBT people to put in the, you know, to, to put as your meme for the day.

    4. JR

      Hmm.

    5. RD

      So there's a lot of, um, there's a lot of sophistication, there's a lot of understanding of American culture, and then there's a lot of understanding of trolling culture, and so these things combine to be a rather effective, uh, you know, very effective social media agency.

    6. JR

      And is there an overwhelming sort of narrative that they're trying to pursue, they're trying to push?

  8. 14:50 – 19:20

    Building ‘tribes’ across platforms: long-game community cultivation

    1. RD

      So what we saw, so I did the, um, I did some of the research for the Senate, and the Senate data came from the platforms. So what I had was, um, the attribution was made by the platforms, it wasn't like Renee deciding this was IRA, it was, uh, the platforms giving it to, uh, to our government. And the information in there, um, what it showed was that across all platforms, across Twitter, across Facebook, Instagram, YouTube, they were building up tribes. So they were really working to create distinct communities of distinct types of Americans and that would be, for example, there's an LGBT page that is very much about LGBT pride, there's, um...

    2. JR

      And they created it.

    3. RD

      And they created it.

    4. JR

      And they f- they curate it and they...

    5. RD

      Create it, curate it, um, it has a, you know, their, it's like a persona, a lot of the posts on the LGBT page were written by what sounded kind of like a millennial lesbian was the voice, um, so it was a lot of, um, you know, memes of LGBT actresses and they would brand it with a specific brand mark, it was a rainbow heart. Um, uh, LGBT United was the name of the page, it had a matching Instagram account, which you would also expect to see from a media property, right? You would expect them to see in both places, and this, um, you know...

    6. JR

      What were they pushing?

    7. RD

      It read like a, it read like a, a young woman talking about, um, crushes on actresses and things, actually. You know, it was, it was, it was really, besides the sometimes wonky English, virtually indistinguishable from what you would read on any kind of like young millennial focused, um, social page. It, it wasn't, uh, none of it was radical or divisive, it wasn't like, um ... the way that they got the division across was they built these tribes where they're reinforcing in-group dynamics. So you have the LGBT page, you have, uh, numerous pages targeting the Black community, that was where they spent most of their energy, a lot of pages targeting, um, far right. So, uh, both old far right, meaning, um, people who are very concerned about what does the future of America look like, and then young far right, which was much more angry, much more like trolling culture. So they recognize that there's a divide there, that the kinds of memes you're gonna use to target younger right-wing audiences are not the same kinds of memes you're gonna use to target older right-wing audiences.

    8. JR

      Mm.

    9. RD

      So there's a tribe for older right wing, younger right wing, in the Black community there's a Baptist tribe, there's a Black liberation tribe, there's a, uh, Black women tribe, there's one for people who have incarcerated spouses, there's a, uh-

    10. JR

      Wow.

    11. RD

      ... Brown, um, Brown Power I believe was the name of it, page, that was very much about, um, Mexican and, uh, Chicano culture. There was Native Americans United. Uh-

    12. JR

      And all of these are fake.

    13. RD

      All these are fake.

    14. JR

      All these are fake. And what are they trying to do with all these?

    15. RD

      So you build up this in-group dynamic and over, and they did this over years, so this was not a, a short-term thing. They started these pages in 2014, 2015 timeframe, most of them. Um, they started some other ones that were much more political later, and we can talk about the election if you want to, but with this tribal thing, um, you're building up tribes so you're saying like, "As Black women in America, this is, um, here's posts about things that we care about. Here's posts about Black hair, here's posts about child rearing, here's posts about, um, fashion and culture." And then every now and then, there would be a post that would reinforce, like, "As Black people, we don't do this and so..." or "As LGBT people, we don't like this." And so you're building this rapport, so like me and you, we're having a conversation, we're developing a relationship on this page over time-And then I say like, "As this kind of person, we don't believe this." So it's a way to subtly influence by appealing to an in-group dynamic or appealing to like, as members of this tribe, as LGBT people, of course we hate Mike Pence.

    16. JR

      Hmm.

    17. RD

      As Black people, of course we're not going to vote because, you know, we hate Hillary Clinton because we hate her husband. As, um, as people who are concerned about the future of America, as Texas secessionists, you know. So, so everything is presented as members of this tribe, we think this. As members of this tribe, we don't think this.

    18. JR

      But a lot of the posts-

    19. RD

      And that's where you see the ... Go ahead.

    20. JR

      Sorry, but a lot of the posts were not even political, they were just sort of affirming the standards of the tribe.

    21. RD

      Yes.

    22. JR

      So they were kind of setting up this whole long game.

    23. RD

      Yep.

  9. 19:20 – 26:41

    Scale and rebranding tactics: sleeper accounts, repurposed pages, and meme experiments

    1. JR

      And then once they got everybody on board, how many followers are these, do these pages have?

    2. RD

      So the, there was kind of a long tail. There were, um, I think 88 pages on Facebook and 133 Instagram accounts, and I would say maybe 30 of the Facebook pages had over 1,000 followers, which is not very many, and then, um, maybe the top ten had upwards of 500,000 followers.

    3. JR

      Hmm.

    4. RD

      So there's, you know, same way you're on any social campaign, sometimes you have hits, sometimes you have flops.

    5. JR

      Right.

    6. RD

      And what was interesting with the flops is you would see them repurpose them.

    7. JR

      Hmm.

    8. RD

      So they would decide, you know, the same way if you're running a social media agency, well, we've got this audience. This page isn't doing so well, let's like rebrand it a little bit, change it up, try to, um, try to make it appeal to somebody else. Like, so you do see this.

    9. JR

      Like what did they do?

    10. RD

      There was a (laughs) there was, um ... I got this data set and I was going through these Instagram memes and, you know, 133,000 of them, and I was, um ... There was a cluster of images of Kermit the Frog. And I was like, "What the hell is Kermit the Frog doing in here?" And so I, so then I go, so this, the (laughs) the way the platforms provide the data is I got like a CSV of the posts and then I got a folder of the images, and so in order to like connect the dots, I had to have the image up on one screen and the, this, um, thing, the CSV up on the other screen.

    11. JR

      CSV?

    12. RD

      Uh, it's like a spreadsheet.

    13. JR

      Okay.

    14. RD

      Yeah. And I, and we, you know, turned it into a database that we could track things a little bit more easily across the platforms. But, um, so I have this cluster of Kermit the Frog memes and I go and I look and I realize that the- they're attributed to an account called Army of Jesus. And I thought, "Well, that's interesting," because what are, you know, these are ... Some of them are really raunchy. (laughs) It was like, it was like Kermit, Miss Piggy, like, you know.

    15. JR

      Mm-hmm.

    16. RD

      I, I mean, it was just like, like stupid, uh, stupid, crappy memes, um, attached to Army of Jesus. I'm like, "What the hell is going on here?" And I keep going through it, hundreds of Kermit memes, and then I get to a post where they say like, um, "This page is owned by Homer Simpson now. Kermit went to jail for being," like, I don't know, they, they made some like, some joke, it was stupid. And all of a sudden, the data set turns into Homer Simpson memes. So again, like this kind of raunchy Homer Simpson culture. Um, and again, it's attributed to Army of Jesus. And then I go through all this and realize that they didn't get to actually making Army of Jesus a Jesus-focused page until like 900 posts in. So the- they just renamed the account at some point. It used to be called Nuts News and then they ... Nuts News was what they called it when it was the Kermit the Frog meme page and then it gets repurposed when they realize Kermit's not doing it, it's not getting the audience they want. Homer Simpson's not getting the audience or engagement they want. And then they pivot over to Jesus and then all of a sudden they start, you know, the likes and, and things start pouring in. So what they're doing is they're actually like either deliberately or they're just creating placeholders. Um, it's kind of a red flag when a brand new account that was created yesterday suddenly starts talking about some highly politically divisive thing or whatever, but if you lay the groundwork and you do it over a period of two years, then somebody who goes and checks to see what the account was, where it came from, how old it is, is gonna see something that was two years old. So it's an opportunity to create almost like sleeper accounts where you create them now and then you activate them, you politicize them, you actually put them to use a couple years in the future. So we saw all kinds of, uh, you know, we, we saw this over and over again. There was a Black Guns Matter account that turned into an Anonymous account at one point. Um, they were pretending to be Anonymous, you know, the hacktivist.

    17. JR

      Mm-hmm.

    18. RD

      Yeah, so, so they repurposed this Black Guns Matter page which just had, it was, it was advocating that Black people buy weapons and, and carry and it was like a pro-Second Amendment page but for the Black community. And then they took that page, when it wasn't getting, I guess, a ton of engagement, and it became, uh, it was called, um, oh gosh, I don't remember the exact name of the Anonymous page and I don't wanna say it was something that's legit, but (laughs) but, uh, they pivoted it into an Anonymous page. Yeah.

    19. JR

      And when they do that, do they go back and repurpose the con- the content of the earlier posts? Do they change? Like, do they edit them?

    20. RD

      That was not clear. That wasn't clear. We didn't get that information from the platforms.

    21. JR

      Hmm.

    22. RD

      Um, there was a lot of stuff that I would've loved to have more insight into. We could see, again, you know, you'd think if you started following an Army of Jesus page and you had all this raunchy Kermit shit from like a year ago-

    23. JR

      Yeah.

    24. RD

      ... that would raise some flags. I would assume that they scrubbed it and restarted, but I don't know.

    25. JR

      You, your podcast with Sam changed how I look at a lot of the pages that I actually follow 'cause I've, I follow some pages like that have classic cars or something like that, and then I'll see them and most of it is just photographs of cars, like beautiful old cars, and it'll have, you know, they'll have a, a giant following, and then all of a sudden something will get political. And I'll, I'll look at it and I'll go, "Oh, wow, like this is probably one of those weird accounts." Like they're getting people to get engaged with it because it represents something that they're interested in, like classic muscle cars, and then they use it for activism or they use it for, to, to get this narrative across.

    26. RD

      I think ...I, I mean, I've seen it happen with some of mine too. Um, (clears throat) I think one of the challenges is like, you want people to be aware that this stuff exists, but you don't want them to be paranoid that it's-

    27. JR

      Right.

    28. RD

      ... everywhere.

    29. JR

      I am paranoid.

    30. RD

      And that's... I know. (laughs)

  10. 26:41 – 33:58

    Election-era objectives: pro-Trump, anti-Clinton, depressing turnout, and polarizing both sides

    1. JR

      Now, uh, the, the overwhelming narrative is that the Russians were very much invested in having Trump win, right? And if they were very much invested in having Trump win, w- was the reason why they focused so heavily on the African American community because the ha- African American community traditionally seems to vote Democrat? So they were trying to do something to break that up or trying to do something to weaken the position of the, the, you know, the incumbent or Hillary Clinton and maybe put some emphasis on Jill Stein or some alternative candidates?

    2. RD

      Yeah, so the, the way that the political campaign, the political aspect of it played out, so they established, they started building these relationships in 2015. And, you know, they're doing this, this tribal thing, "We've got our in group, we're part of this community." And then what you start to see them do is, um, early... There actually, there was a tiny, tiny cluster in the early primaries where they were supporting Rand Paul, and then they pivoted Trump pretty quickly, and he was... And probably Rand Paul just didn't poll well and they were like, "There's no way to get any lift here," but, but maybe Trump was getting, you know, some actual lift in the media. And so, so you see them move into supporting Trump, and then for the remainder of the dataset for, from 2015 through the end, which was mid-2017 or so is when this thing ends, um, it's, it's adamantly pro-Trump on the right. And on the right, you see not only pro-Trump, but you see them really working to, like, erode support for mainstream or traditional Republicans, traditional conservatives. You see a lot of the memes about like, um, "Are you with the cuckservatives or the conservatives?" And they're, you know-

    3. JR

      Hmm.

    4. RD

      And so the cuckservatives, of course, are like... They've got pictures of, like, Lindsey Graham and John McCain. They hate John McCain. John McCain shows up a million times. Um, and, uh, Marco Rubio-

    5. JR

      Is it clear why?

    6. RD

      ... Ted Cruz. Well, I think that they, you know... One of the theories is, um, and, and I believe this is probably true, they, they really strongly disliked Hillary Clinton because there was concern that she would, you know... Things that she was saying about increasing freedoms in Russia were very threatening, uh, they thought the best bet to get sanctions removed was Trump. Um, so they had specific outcomes that they were hoping for, and that was one of... You know, so there's always, like, a political motivation. So there is this narrative around they just wanna, kind of like, screw with American society, create divisions, amplify divisions. When you look at the political content, the clear and c- and sustained support for Trump, and even more than that, the clear disdain for Hillary Clinton. There is not... On Facebook and Instagram, there was not one single pro-Hillary post. There were some anti-Trump posts, because if you're running an LGBT page, of course they're gonna say negative things about Trump, you know? And, and they're saying it, "So you should vote for Jill Stein."

    7. JR

      Hmm.

    8. RD

      There was early support, um, on some of the left-leaning pages for Bernie Sanders, but you actually see the support for Bernie Sanders come in more after it becomes clear that he's not going to win, because then they're using Bernie Sanders as a way to say, "This was stolen from him by the evil Clintons," um, or, "Jill Stein, you know, here's a true, a true independent, real liberal. Uh, we should be voting for her if we want to support a woman." So there are these feminism pages really pushing this narrative of Jill Stein. So you have the left-leaning pages, totally anti-Clinton, and then you have the right-leaning pages, staunchly pro-Trump and also strongly anti-Cruz, anti-Rubio, uh, anti-Lindsey Graham, basically anti every now what's called "establishment" Republican. Um, and there's this, uh, kind of pushing of people to opposite, opposite ends of the political spectrum. This is where you get at the conversation around facilitating polarization. So not just... Um, it, it wasn't enough to just support Donald Trump, it was also necessary to strongly disparage, um, the kind of traditional conservative, moderate center right in the course of amplifying the Trump candidacy.Does that make sense?

    9. JR

      Yes, it does.

    10. RD

      I know it's a lot of stuff.

    11. JR

      Yeah, it is a lot of stuff but, i- it does make sense and one of the things that was really bizarre to me watching the election, a- and I was trying to figure out, is this because Trump is so bombastic and he's so outrageous and he's just a different person, that the- the way I was describing it on stage was that like finally the assholes have a king.

    12. RD

      (laughs)

    13. JR

      Because they never had a king before. Like everyone who was running for president was at least mostly dignified. I mean, b- basically, it's really difficult to go back in time and find someone who isn't.

    14. RD

      Yeah.

    15. JR

      Find someone who d- there's no one who insults people like he does. I mean, he insults people's appearances, he calls them losers, he called Stormy Daniels horse face. I mean, he- he says some outrageous shit. So part of it was me thinking like, "Wow, maybe he's just ignited an emboldened..." I actually had this conversation with my wife today. She was like, uh, "I've- it feels like racism is more prevalent." Like it's more, it's more accepted. People feel more emboldened, because they're- in their mind, they think he is a racist, "I can get away with more things, Trump is president." Like there's actually videos of people saying racist shit and saying, "Hey, Trump's president now. We can do this." So, I was thinking that, well maybe that's what it was, it's just sort of like some rare flower that only blooms under the right conditions, poof, it's back, right? But when you think about the influence that these pages have had in establishing communities and this long game that they're playing, like the LGBT pages, even though they're shitting on Trump, they really wanna support Jill Stein because they know that'll actually help Trump because it'll take votes away from Hillary Clinton. That- they m- it seems different. Like political discourse, discussions online and social media, the way social media reacted, I mean there was a lot of people that were anti-Obama, uh, before, you know, m- either- e- either of his elections that he won. But it seemed different. It seemed different to me than this one. This one seemed like, l- like we had moved into another level of hostility that I'd never experienced before, and another level of division between the right and the left that I'd never experienced before. And, uh, a- uh, b- like a willingness to engage with really harsh, nasty comments, and just to- to dive into it. You would see it all day. I mean there's- there were certain Twitter followers that I think they're pretty much human beings, but I would follow them and they would just be engaged with people all day long. Just shitting on people and criticizing this and insulting that, and i- it seemed like, it seemed dangerous. It seemed like things had moved into a much more aggressive, uh, much more hostile and confrontational sort of chapter in American history. If this was all done at the same time as it's happening, w- how much of an influence do you think this IRA agency had on all this stuff?

  11. 33:58 – 39:29

    Measuring impact is hard: engagement stats, missing comment context, and platform differences

    1. RD

      That's the, uh, that's the question that we would all like the answer to and I unfortunately can't give it.

    2. JR

      Yeah.

    3. RD

      And so, let me-

    4. JR

      In your mind though.

    5. RD

      Yeah, let me, let me, let me kinda caveat that. Um, the thing that we don't have, that nobody who looks at this on the outside has, is we can't see what people said in response to this stuff.

    6. JR

      Mm-hmm.

    7. RD

      So I've looked at now almost t- 200,000 of these posts. Um, that's what I spent most of last year doing, was- was this- was this research. And we can see that they have thousands of engagements, thousands of comments, thousands of shares. We have no idea what happened afterwards, and that's the problem. So when- once the stuff comes down, it's really hard to go back and piece it together. So I can see that there are some of, per your point, the- the really, really just fucking horrible troll accounts that they ran. They didn't necessarily have a lot of followers but you see them in there, like, @ing people. So they're, you know, @ and then the name of a reporter-

    8. JR

      Mm-hmm.

    9. RD

      ... @ the name of a prominent person and so they're in their kinda like draft on the popularity of, you know, famous people basically.

    10. JR

      Mm-hmm.

    11. RD

      Um, and they're just saying, like, horrible shit. (laughs) And- and it's- the tone is so spot on. And one thing that was interesting with a couple of them is like if you go and you look at their profile information, which was also made public, they would have like, um, they would have a b- uh, like a Gab account in their- in their profile that was like, so they would, um... So it was a remarkable piece of- of- of kind of the culture in which you see that like, they're actually sitting on Gab too, right? And- and so they can also go and they can draw on, they're in Reddit, there's you know, 900 or something, uh, troll accounts were found on Reddit, they're on Tumbler. And so they're just picking the most divisive content and they're pushing it out, uh, into communities and at the same time we can see that they're doing it, but we can't see what people do in return. We can't see did they just block? Did they have to fight back? Did-

    12. JR

      Right.

    13. RD

      Was there a huge, you know, when this happens on a Facebook page, um, and they're doing something like telling black people not to vote, um, "As black people we shouldn't vote," um, what do people say in response? And that's the piece that we don't have. So when we talk about impact, a lot of the impact conversation is really, uh, focused on did this swing the election? We don't have, nothing that I've seen has the answer to that question. Um, the other thing is, yeah but- but the second question, the thing when I think about impact, I think from- from, I think you and I agree on this, um, it also matters how does this change how people relate to each other? And we have no real evidence of, that we have no information on that either. This is the kind of thing that lives in some, you know, Facebook has it, the rest of us haven't seen it.

    14. JR

      Now are most of these people, is this mostly Facebook? Is it mostly Twitter? Uh where- what does- how does it break down?

    15. RD

      Yeah, so there were, um... Here are my like little stats here 'cause I didn't want to give you the wrong data. There were 10.5 million tweets of which about six million were original content created by about 3,800 accounts. Um-There were about 133, let me just read it, 133, um, Instagram accounts with about 116,000 posts, and then 81 Facebook pages and 17 YouTube channels with about 1,100, um, videos. And so they got about 200 million engagements on Instagram and about another 75 million or so on Facebook. Um, engagements are like, likes, shares, comments, reactions, you know. Um, so it's hard to contextualize, but what we think happened, you know, you can go and you can try to look at, um, how well did this content perform relative to other real authentic media targeting these communities? And what you see with the Black community in particular is their Instagram game was really good. Um, their... so their... on their Instagram accounts, the, you know, the top five, three of them targeted the Black community and got, you know, um, 10s to 100 millions of engagements.

    16. JR

      Wow.

    17. RD

      So I would have to pull up the exact number-

    18. JR

      Is it mostly memes?

    19. RD

      I don't know it off the top of my head. Yeah, it's um, it's... on Instagram it's all memes. And then, you know, so we have the memes and then we have the text. On Instagram you can't really share, so it's amazing that they got the kind of engagement that they did even without the sharing function.

    20. JR

      Mm-hmm.

    21. RD

      But one of the things you can do is if you know the names of the accounts and they're... a lot of them are out there publicly now, um, you can actually see them in regram apps. So, uh, people were regramming the content. So Facebook says about 20 million people, excuse me, engaged with the Instagram content. But what isn't included in that is all of the regrams of the content that were shared by other accounts. So the spread and the dispersion of this, it's an interesting, uh, an interesting thing to try to quantify because we have engagement data, but we don't know did it change hearts and minds? We don't know if it-

    22. JR

      Right.

    23. RD

      ... um, influenced people to go follow other accounts. We don't know if it influenced people to not vote. Uh, there's just so much more I think still to, to understand about how these operations work.

    24. JR

      Well, we can kind of, we can assume that it had some impact, right? I mean-

    25. RD

      Yes.

    26. JR

      ... as, as you were saying earlier, when a new person enters into a conversation, it changes the tone of it. How much of what they did was their own original post and how much of it was commenting on other people's posts?

  12. 39:29 – 56:00

    Repurposing American content: copying local news, memes, and native vernacular to avoid detection

    1. RD

      So I thought you were actually gonna ask a different thing there (laughs) .

    2. JR

      Oh, please. Well, what did you think I was gonna ask?

    3. RD

      There's the, how much of it was them repurposing our own posts, right. Repurposing real American content.

    4. JR

      Hmm.

    5. RD

      Um-

    6. JR

      Did they do that as well?

    7. RD

      Yeah, tons of times.

    8. JR

      Hmm.

    9. RD

      Uh, but let me, let me, uh... so they created a lot of their own stuff, particularly in the early days. And so you can actually read the data set, and one of the things when we started finding these posts, I was struck by how sometimes it read like ESL, and then sometimes it read like perfect flawless, um, professional English. And then other times it read like normal English, vernacular, just the way that we would talk to each other. And I started digging into what that was. So when it was vernacular English, when it was... when it, when it read like fluent American, uh, American English, it was, um, usually cribbed from somewhere else. So they would go and they would find a local news story from some obscure local paper, and they would crib, and then they would paste that. And then, so the Facebook post would be that cribbed, uh, sentence from that article, and then their meme. Uh, maybe they would add a sentence underneath it to give it some kind of context or angle. When they would write their own stuff, you would see the sloppiness. That's where you could see, um, subject verb agreements not quite there. The, you know, ways in which like Russian possessives are different than American possessives, the slips there. Um, and then the other thing was the really funny stuff, which was, um, you know, a, a post that's supposed to... supposedly written by Texas secessionist, right? So you can probably have an image of a Texas secessionist in your mind as I say this. And it would be things like, um, "Hillary Clinton is a terrible individual, and as a, uh, as a terrible individual, it's, uh, completely impossible for us to back her in her candidacy for the American presidency. Furthermore..." You know, it's like, furthermore, ergo, you know.

    10. JR

      Right.

    11. RD

      It is clear that... and I'm like, it reads, like remember you're like in, you know, English in college or something, you've got to like write-

    12. JR

      Yeah.

    13. RD

      ... a formal essay. It was like, okay-

    14. JR

      Right.

    15. RD

      ... come on.

    16. JR

      You bullshit your way through it.

    17. RD

      Right.

    18. JR

      Yeah.

    19. RD

      So nobody actually talks like this, especially not, you know, your, your-

    20. JR

      Texas secessionist.

    21. RD

      ... stereotypical Tex- Texas secessionist. So it was funny seeing these, um, incongruities. And that's unfortunately, one of the best ways to tell what you're dealing with is actually to kind of look for those incongruities now and, and see, uh, as you read communications online, like can... you know, does this, does this read like an American? Does this read like a communication? And what we started to see was one way to not get caught for your lousy English or your, you know, your cultural, um, lack of, uh, kind of native, native, um, abilities, is to just repurpose other people's stuff. And so that's where you would see memes getting shared from, uh, on both the right and the left. You know, you'd see a lot of these like Turning Point USA memes that they were repurposing and pushing out. Or you would see, um, Occupy Democrats or the Other 98%. So memes from real American pages, real American culture, and they would just sometimes slap a new logo on and just repost it as if it was theirs. So it does, in those instances, read just like, you know, authentic American content. And in many ways it is authentic American content.

    22. JR

      How many people are working for this agency?

    23. RD

      It's a-

    24. JR

      Do we understand?

    25. RD

      I don't remember. Off the top of my head it was, um, somewhere between a couple hundred and a thousand, I think. I don't know if it's bigger than that now.

    26. JR

      And they're just-

    27. RD

      They just moved-

    28. JR

      ... constantly-

    29. RD

      ... offices and, um ... (laughs) This is a funny story. I guess. They, uh, moved offices and then, um, people started calling in bomb threats to the office, and it was just like every day a new bomb threat would get called in, so they couldn't work basically.

    30. JR

      (laughs)

  13. 56:00 – 1:14:03

    From online posts to offline action: funding activists, creating protests, and staged confrontations

    1. RD

      But what you see in the, um, in the Mueller indictment is the text messages, is the, the messenger, the Facebook Messenger messages, where they're going back and forth with real activists and they're saying things like, um, you know, "Hey, my ad account got shut down. Can you run some ads for me?" (laughs) Or the, um, "Hey, I wanna help your protest. Um, we're fellow Black Lives Matter activists and we see you're running a protest up in ..." and I think it was like Ithaca or something, um, Binghamton. "How can we help you? We can give you some money for posters." And they're sending money for posters. Or they reach out to a Trump supporter and they say like, "We think it'd be really funny to have a Hillary Clinton impersonator sitting in a truck, flatbed truck, that's made up to look like a jail. Let us, um, you know, if we give you some money, will you find the Hillary Clinton impersonator and, um, put her in jail and do this Hillary for prison thing?" And so this is where ... Or another thing that they did was, um, using Facebook events to create real world protests. So they're not limiting it to shit posting online and making people feel tension online, they're actually sending people out into the real world to have in street violence. And so one of the things that they did was they coordinated a Facebook event, um, one for the Texas secessionist page and one for the, there was a pro-Muslim called United Muslims. And on the same day at the same time in Texas, they have a rally to support, um, Texas culture and resist the Islamicization of Texas across the street from a rally to defend Muslim culture. And so they, like there's literally no-

    2. JR

      (laughs)

    3. RD

      ... you know, th- they just create these Facebook events on these pages and, and then they promote them with ad dollars and other things. And, and you literally, if you go and you look at the Texas reporting from that day, um, I don't remember if it was dozens or hundreds, but a, a sufficient number of people showed up that they had literally on opposite sides of barricades, police officers in the center screaming at each other, because one group is there for the resist the Islamicization of Texas and the other group is there to like defend Muslim culture. And so you get two, you know, two opposite sides of the spectrum in the same place at the same time, and you literally incite like a mini riot. So they, there, there were about 81 of these events where they were holding, um, Black Lives Matter style rallies for victims of violence, police violence, um, memorials for people who were killed by police officers. You know, things that real Americans would do, but this wasn't being done by real Americans. And that's, that's the insidious thing, right? Is, is how does, how does Facebook detect that? How do you know? How do you, when you see this come to defend Texas culture and you're a, you know, diehard proud Texan, you know, you're not thinking like somebody in St. Petersburg is, is, is organizing this. And that's the, I mean, I think that the idea that this was just some memes, um, is just not, it, it, it doesn't respect the significance of what they were trying to do and how effective that they were with these other things. Or even if they're just trying it out, just a little bit, just working to see what works, they're always experimenting, they're always trying to find ways to create that tension. Um, and that's the thing that I think is so interesting about this, right? This evolving, this idea of an information war where these tactics evolve, um, and you are really at a disadvantage when it comes to actually detecting them.

    4. JR

      Yeah. And it, and on the outside, if you're looking at that, you'd say, "Well, okay, what is their objective?" Why would they have this Texas secessionist page across or rally across the street from a pro-Islam rally? Why would they do that? You know, if you're on the outside, you think about the amount-

    5. RD

      (coughs)

    6. JR

      ... of effort that's involved in doing something like this, and they're also doing this with no leaders, right?

    7. RD

      Right. There's no one there.

    8. JR

      There's no one there.

    9. RD

      So all the pro-Texas people go, "Here we are."

    10. JR

      (laughs)

    11. RD

      "Look at over there. It's the motherfucking enemy."

    12. JR

      (laughs)

    13. RD

      (laughs) I think a couple times there were comments, um, on some of the like archived pages and things where you could see the screenshots of people being like, "Dude, you hel- us all come out there and like nobody showed up."

    14. JR

      Right. (laughs)

    15. RD

      (laughs)

    16. JR

      Well ...

    17. RD

      Who is in charge? You know? (coughs)

    18. JR

      They're s- they're probably throwing a lot of things against the wall hoping that they stick.

    19. RD

      Hoping something sticks. Well, you see this on the, um ... There was a page called Black Matters, and Black Matters was interesting 'cause they went and made a whole website. So they made a website, blackmattersus.com, which I think is still active. It's dormant, they're not updating it, but I believe you can go and, and read it. Um, and it was designed to call attention to police brutality type things. And so they have this Black Matters US page, and then there's the Black Matters Facebook page, the Twitter account, the Instagram page, the YouTube channel, the SoundCloud podcast. (laughs)

    20. JR

      Wow.

    21. RD

      The Tumblr, the s- Facebook stickers. They had Facebook stickers that looked like little black panthers, like little, um-

    22. JR

      Really?

    23. RD

      ... little cats. Yeah, little black cats. They were actually really cute, very well done. Um, so you have this entire fake media ecosystem that they've just created out of whole cloth, all theirs. And then what they start to do is they start to put up job ads-And so it's, "Come be a designer for us. Come be a, uh, come write for us. Come photograph our protests. Come ..." Um, you know, they, they had like a kind of like, um, Black guy dressed in a cool outfit, like hipster, you know, holding a sign like, "Join Black Matters." Um, you see them go through a couple different logos the same way you would if you were starting a, you know, media brand. Um, they start posting ads for, "Do you want to be a calendar girl? Send us your photos. Uh, do you want to be on a Black reality TV show? Send us video clips of you." Um, do you ... Like, they begin to do real work to ingratiate themselves with the community. They had a physical fitness thing, it was called Black Fist and the idea was that it was, um, kind of vaguely militant-esque, um, in that it was supposed to teach, uh, Black people how to handle themselves at protests should there be police violence, how to fight back. And they actually went and found a guy, a physical fitness, you know, a, a martial arts, um, guy and they were paying him via PayPal. So (laughs) so, so he was running classes for the Black community under this Black Fist, um, brand, and they would like text him or call him. He, he played some of the voicemails on TV, actually I heard them. Um, (coughs) after my report came out, I think they tracked him down and, um, and he just talks about how they, yeah, they just PayPaled him, you know, a couple hundred bucks every time he ran a fitness class.

    24. JR

      What were the voicemails like?

    25. RD

      It was, um-

    26. JR

      "Hello. We are fellow Black men-"

    27. RD

      (laughs)

    28. JR

      "... concerned about police."

    29. RD

      They, you'd, you'd be (laughs) you'd be surprised. They actually had a YouTube channel with, um, two Black men named Williams and Calvin and there was this channel, Williams and Calvin, and-

    30. JR

      They were actual Black men?

  14. 1:14:03 – 1:56:46

    Platform accountability, moderation debates, and regulation: Section 230, ads transparency, and privacy

    1. JR

      Colds are going around. Don't feel bad. I'll get you one. Hold on a second. Um, who's, who's seeing all this stuff? Is this stuff-

    2. RD

      What do you mean?

    3. JR

      Uh, has ... Obviously Facebook has, uh, checked this out. I'm sure Twitter's aware.

    4. RD

      Mm-hmm.

    5. JR

      What has the reaction been and is there any sort of a concerted effort to mitigate some of, some of the impact that these, these sites have?

    6. RD

      Yeah. Lots of it, actually. So I think, um, in, in 2017, um, was when we started, like, "we" being independent researchers, I guess, people on the outside of the companies, academics, um, began to find the content, you know, really began to... Investigative journalists would identify the name of a page and then me and people like me would go and we would scour the internet looking for evidence of what was on that page. So I found a bunch of the stuff on Pinterest, for example, wrote about it. Um, a guy by the name of Jonathan Albright, uh, found a CrowdTangle data cache and with that we got the names of a bunch more pages, bunch of more posts, and we had some really interesting stuff to work with. Originally, the platforms were very resistant to the idea that this had happened. (coughs) And so as a result of that, they were, um, in, you know ... There was a, the first thing that Zuck said in 2016 when, you know, Trump gets elected, Twitter goes crazy that night with people who work at Twitter saying, "Oh my God. Were we responsible for this?" Which is (laughs) a very Silicon Valley thing to say. Um, but what I think they meant by that was, their platform had been implicated as hosting Russian bots and fake news and harassment mobs and a number of other things, and there was always the sense that it didn't have an impact and it didn't matter. And so this was the first time that they started to ask the question, "Did it matter?" (coughs) And then Zuck made that statement, um, fake news is a very small percentage of, you know, whatever, on Facebook, the amount of information on Facebook, and the idea that it could have swung an election was ludicrous. So you have the platforms kind of, uh, the leaders of the platforms digging in and saying it's inconceivable that, that, that this, you know, could have happened. And as the research and the discovery begins to take place over the next nine months or so, you st- you, you get to when the tech hearings happen. So I worked with, um, a guy by the name of Tristan Harris. He's the one who introduced me to Sam, um, and he, he and I started, uh, going to DC with a third fellow, Roger McNamee, and saying, "Hey, there's so much ... There's this body of evidence that's coming out here and we need to have a hearing. We need to have Congress ask the tech companies to account for what happened, to tell the American people what happened." Because what we're seeing here as outside researchers, when investigative journalists are writing, the things that we're finding just don't line up with the statements that, that nothing happened and th- this was all no big deal. And so we start asking (coughs) for these hearings and actually, um, myself and a couple of others then begin asking them in the course of these hearings, "Can you get them to give you the data?" Because the platforms hadn't given the data. So it was that lobbying by concerned citizens and journalists and researchers saying, "We have to have some accountability here. We have to have the platforms account for what happened. They have to tell people." Um, because this had become such a politically divisive issue. Did it even happen? And we felt like having them actually sit there in front of Congress and account for it would be the first step towards, um, towards moving forward in a way, but, but also towards, um, changing the minds of the public and making them realize that what happened on social platforms matters. And it was, it was really interesting to, to, to be part of that as it, as it played out, um, because one of the things that Senator Blumenthal, one of the, one of the senators did was actually said, um, "Facebook and Twitter have to notify people who engage with this content." And so the- there was this idea that, um, if you are engaging with propagandist content, you should have the right to know.

Episode duration: 2:07:54

Install uListen for AI-powered chat & search across the full episode — Get Full Transcript

Transcript of episode UAGZcGi1OP8

Get more out of YouTube videos.

High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.

iOSAndroidClaudeChrome