[Interview] Mark Ledwich - Algorithmic Extremism: Examining YouTube's Rabbit Hole of Radicalization
machine learning youtube recommendation algorithm extremism alt right pipeline pathway mainstream radicalization
Interview with one of the authors of a widely reported study on YouTube's recommendation engine and where it leads its users.

https://arxiv.org/abs/1912.11211
https://www.recfluence.net/
https://github.com/markledwich2/Recfluence
https://www.patreon.com/ledwich

Abstract:
The role that YouTube and its behind-the-scenes recommendation algorithm plays in encouraging online radicalization has been suggested by both journalists and academics alike. This study directly quantifies these claims by examining the role that YouTube's algorithm plays in suggesting radicalized content. After categorizing nearly 800 political channels, we were able to differentiate between political schemas in order to analyze the algorithm traffic flows out and between each group. After conducting a detailed analysis of recommendations received by each channel type, we refute the popular radicalization claims. To the contrary, these data suggest that YouTube's recommendation algorithm actively discourages viewers from visiting radicalizing or extremist content. Instead, the algorithm is shown to favor mainstream media and cable news content over independent YouTube channels with slant towards left-leaning or politically neutral channels. Our study thus suggests that YouTube's recommendation algorithm fails to promote inflammatory or radicalized content, as previously claimed by several outlets.

Authors: Mark Ledwich, Anna Zaitsev
 Alright, I'm very pleased to have Mark Ladoitch here today in In he's the he's one of the authors of this paper. That's called algorithmic extremism examining YouTube's rabbit hole of radicalization So I've done a video about a topic like this before actually several and this is basically one in a line of research that examines the recommendation algorithm of YouTube specifically but also kind of the general Social media platforms. So Mark, thanks for being here Could you maybe for people who do not know anything about this could you kind of Explain where your work fits into what's been done before or kind of also what comes out of the of the mainstream Media about this topic because there's been quite a bit of of talk Yeah, so I'm not a researcher by trade I'm a programmer and the reason why I got into this was because I could see clear bias in the way The YouTube recommendation system is being reported on and also in the research a There's some narratives. I think it might be because There's a lot of people worried about rhyming populism and this is a way to explain that They're looking for ways YouTube are radicalizing people and finding evidence for that or But that could be anecdotes or in some of the studies at sexual quantitative data But they're only looking to confirm it. So there's really obvious things. I think you covered in your video Some of them will just look for movement towards alright channels Through like centrist Or alt-light they call it instead of looking for both ways. Just really obvious things like that Calling it calling it an infection that cliche clearly shows that really looked at it Like a curious person would so I thought I could easily as a software engineer just collect all the data And without any complicated statistics just looking at the overall Flow of recommendations between the two the overall flow of recommendations between videos What their political influences? Yeah, this this was a thing that that bugged me of the paper that I made a video about is that they claim there's this radicalization pipeline, right and with pipeline everyone sort of understands a different thing But I think the general consensus is that the recommendation algorithm itself will steer you towards like more extreme content and in this case towards like the alt-right extremist content and the paper actually analyzed and said Okay, we found evidence that there is movement in this direction But they never shown that this is significantly more movement than like in in the other direction So in order to justify a pipeline one would need to show that the movement this way is about larger than this way in some notion and So I've I've found I've actually spoken to the author of that paper and he agrees with that but Obviously doesn't Doesn't have like energy to go into every you know go Refute everything that comes at them. They've also been a bunch of like they've also been exposed to a lot of Criticism, let's say as have you and I think even more when when your paper came out, I think The four days there was just a giant storm of people Attacking your paper basically Basically just just listing every single thing that's wrong with it and why this isn't valid and Things like this. So let's actually jump into what you did specifically so if I'm if I'm can summarize and you can then maybe correct so that we can Establish what happened you basically collected? recommendations, so you scrape these videos on YouTube and you collected these recommendations and We can we can see this so in your paper you then can make such diagrams Such as this one or these where in the middle the white bar is a Channel or a group that you're interested in and then to the left you can see where all the impressions of that Channel or group come from so what's where where basically the views come from? Through the recommendation system and on the right you can see of all the views the channel has retrieved Where do they go to so what what what is recommended next? Right, so it basically shows both directions for for every group and then you've also labeled these by multiple methods so that you can kind of establish these groups and What is pretty cool? We've built this website where you can analyze this and my computer is a bit overloaded at the moment But I promise it's really interactive. All right, so during the interview my computer crashed So I'm doing this in post-production Just to show you how this website operates So what you have here is an overview over all the channels of what rare recommendations were collected And they are grouped into groups for example here after partisan left Center left social justice partisan right and so on so you can see for each group or channel where recommendations come from and where they go to For example the large red one here. I happen to know that is Fox News You can see Fox News of the daily impression it received from itself 36 million impressions and it gives to itself 36 million these numbers have to agree by nature of how the data is collected of course But you can also see it gets 2.7 million impressions from CNN 2.6 million from the next news network and so on and it also gives quite a bit of recommendations to CNN and so on so You can go for example at some individual channel. Here's the daily wire the daily wire is Mainly run by Ben Shapiro So it's a bit more to the right of Fox News and a bit more on the direction of alternative media You can see the daily wire gets some most of its impression Count wise from itself from the daily wire, but it gives most of them to Fox News So actually you can see here that itself is a long way down Like in whatever sixth or seventh place So actually if you were to watch the daily wire the recommendation system would most likely steer you towards something like Fox News Whereas the the claim is that the YouTube algorithm would actually steer you towards more radical content Actually in in reality, it seems like it would steer towards more of these mainstream content So actually want to go to this tab you can see different groupings here and The radicalization pathways is the previous paper we have looked at So they have all these channels here of this radicalization pathway and you can see here the control group gives very very very few Impressions to the IDW The IDW gives much more impressions to the control group, right? Again, and the IDW gives very few impressions to the alt light compared to the amount of Impressions the alt light gives to the IDW and even to the control group And if you look at the alt right and we're going to zoom in on that here It's even more so the alt right of course receives most of its impressions from itself Which you could expect for any kind of group. This is your classic filter bubble situation But if we analyze the question of is there a pipeline you can see that Next most likely you are diverted to the IDW and to the control group much more Than you come from the IDW or the control group, right? Let's look at the the alt light so called this is kind of the so called gateway to the control group So called gateway to the alt right you can see here the alt light gives most of its impressions next to itself To the control group and the IDW so deradicalizing If you look at its way to the alt right, you'll see that it gets about four times as much impressions From the alt right as it gives to the alt right. So Basically, it's kind of taking the steam out of a quarter of all of these sessions and gives it To either the control group or the IDW or itself So this is exactly the opposite of what you would expect If you were to claim that there is a pipeline You would expect their most recommendations to come from more moderate content and go towards more extreme content But it's exactly the opposite and again, these are the exact channels that this original paper used Now what this paper find that the one that we're discussing if you go to media type here What you'll be able to see is the division into mainstream media youtube creator and so-called missing link media Which we'll leave out for a moment Let's focus on mainstream versus youtube creators. You can see the mainstream media gives most recommendations to itself While giving only very little recommendations to youtube creators and the missing link media While the youtube creators actually give almost half of their impressions. Look at that They they like give almost half of their impressions to the mainstream media Which means that there is a big big push by the algorithm to Towards these mainstream media away from youtube creators. So in general and I invite you to look at this website In general, you can pretty much see that the exact opposite of a radicalization pipeline is happening if you of course if you look at these recommendations and how they are distributed actually Most recommendation pathways are towards moderate centrist content and of course creating creating filter bubbles Which is a problem by itself, but is not a radicalization pipeline Lastly, I want to look at white identitarians because it's a one of the groups that people are referring to when they Claim that there are these radicalization pipelines. Look at that So of the white identitarian they get most of their impressions, of course from other white identitarian Videos which would be your filter bubble phenomenon But they give most and this is a group right the white identitarian channels give most of their Recommendations to the partisan right to the central and left mass mainstream media libertarians and and so on and uh Themselves are like really really really far down So to claim that there is this radicalization pipeline if you look at this data to me Seems not justified from this data and if I look at the other paper That really left out the important analysis Of the the backwards direction It seems that given everything it seems that the claim is not warranted All right back to the interview. Um Is that about like what you've done is that a good summary of of the data collection and analysis Um, there's a yeah, it's a good summary I can go into detail. Yeah, please Um, so youtube doesn't make it easy so I started this back in november in 2018 And I was using the youtube api And to get enough uh to get enough quota because they limit the amount of requests you can actually make to their api I created multiple keys, which is against their um policy Um, and they also asked you to delete all your data after 30 days That's also part of their policy. So um later about I think it was october 2019 they cut off my access because I was doing that So I had to move to just uh scraping websites and now My collection process actually just loads up the website and gets the recommendations from the actual page like a user would Um And that's difficult because they block access after a couple of hundred requests. They'll They'll stop you that machine from actually requesting from the website So I need to Use a proxy service that That's fairly expensive and what they do is they simulate or they have actual residential connections through your home connection like atnt and my requests get tunneled through that like a variety of locations in the states to get um A representative kind of sample Cool so so the data collection is Would you say that's that's the hardest part? I feel the labeling of channels is also not so easy But you've you've managed to kind of do that Half automated also half collecting things from kind of um sources that analyze these channels But at least for for most of the things that i've inspected I found the labeling to be pretty sane I think this is always something you can attack the the original paper was also attacked on how they label I find this to be kind of vicarish Mostly I think your labels are pretty good as well. The other papers labels are also mostly pretty okay Yeah, so let's let's go to it. Sorry Yeah, it's quite subjective I expected the labeling to be what I get my pushback on but it turns out it was um the anonymous collection So what you've actually found here what are what would you say are your your main results and I can maybe Show So you've analyzed a bit where do things come from where do things go to and I found this this part here to be One of the even though it's pretty simple One of the core things to say about this is that mostly what you found could be said is It's simply a recommendation algorithm working as a recommendation algorithm should which means it creates You know your typical filter bubbles if you if I watch one minute of this video All of a sudden my site is filled with makeup tutorials and things like this But also you found that there is quite some Over the top push towards what could be considered mainstream media and there is A bit of a draw away from the smaller YouTuber like channels is that is that something that like is that character? I don't know That's right. So it yeah, that's a good way to characterize it if that chart we're looking at now If it was a neutral algorithm The green bars would be the same as the gray ones. So you you receive the same amount of recommendations as you give That would be proportional to the views that you get the future organically The recommendations that you get from the green bars That you get the future organically. Um the recommendations you receive be equivalent to that but we find that it disproportionately recommends mainstream media channels That's not even though. So it's not like um, it doesn't look like it's consistently doing that So you can find exceptions to that rule is um, I I believe one of the main criticisms of your paper has been that you Only use data from 2019 onwards and I have actually looked at your website and your website a lot of times says that the data has been collected from way earlier than that um, so is it that you've almost only used 2019 data in your paper or what is in in the pipe the pipe is just from um november and december 2019 and the reason we did that um Is that we only had 400 channels before that And the collection process have changed over time So this is a clean set of data we could look at and I thought the most recent was the most relevant So what is it doing now? But um, i've provided i've got the same analysis over time So i've got a gift that I made that everyone can look at which goes through all the months i've been collecting Um, and you can see that chart for where it goes to and has gone through a bunch of changes so in about april 2019 That's when they really clamped down on conspiracies and other fringe channels Before that was it was much closer to neutral Okay, so but it never it never looked like a a rabbit hole it's never favoring Fringe channels. Yeah. I mean that that has been my experience also personally on youtube. I've I've joined youtube very early or i've i've watched youtube very early when Young earth creationism was still active and then these things were kind of completely discredited by simply having you having People exposed to other points of view and even I find this now Even though youtube makes it kind of easy to find let's say niche content It also exposes you to a bunch of of different views. Um, and and I've always found this to be very very optimistic in the sense of This is probably deradicalizing much more people than radicalizing But you've you've received like as I said a bunch of criticism in so if you could What was the The largest criticism irrespective of whether it was valid or not. What do you have you found was kind of what most people? were criticizing Most people criticizing that we were collecting anonymous recommendations. It wasn't the personalized ones Yeah, and it's actually like it is a valid limitation. We had it. There's a first limitation we talked about in this paper And It's still an open question How personalization would affect these aggregate results that we've got but I think it's reasonable To assume it will be quite similar once you average it out. So for any one person it might be different But you would expect personalization based on someone's history to even out because It's kind of the algorithms kind of like the average of all that when it's anonymous Yeah, I feel like you'll get that the the the notion of the the the notion that If because if you're not logged in the recommendation is like a person with only one video of history, right? So it's it's the same thing, but there's only one hit point of history instead of multiple I find Why should why should the behavior be qualitatively different if you have multiple points of history? like this is a strong claim that you have to you'd have to really show that there is a qualitative difference not just a more or less accuracy and I feel the people making this criticism are it's really on them to show that there is a substantial difference rather than saying that this is a giant limitation of the work Yeah, and it's also very hypocritical for a lot of the people saying it because some of them like Zion out who was mockingly saying that her article her original article in New York Times Used algo transparency, which is anonymous as well, but she doesn't she never looked into that I think a lot of this is completely motivated reasoning. They don't they don't care about the details I've I've seen this one this one twitter user She she comment she said something to the effect of if you've seen this article, please consult someone that works in this space like it's It's please don't don't read the article yourself. You must you must get your information through someone I've actually i've read the article I've I find it's pretty straightforward the limitations are clear But also the the results are pretty clear and it's it's actually mostly a boring article, right if if I'm sorry, like it's not a criticism. This is good. Like it's mostly you find that things work as expected There is a bit of a push towards mainstream which can be probably explaining that youtube wants to be Advertiser friendly right and these mainstream channels are already are Advertiser friendly so they probably get bumped a bit. Um, if what would you say is Maybe the most the most valid criticism that you've heard maybe not the biggest but the most Where do you where you say? Yeah, this is really This is really something that is you know I think um, I guess what's Um, there was criticism that i'm overclaiming not in the paper so much but in my tweets and medium I guess that's that's fair But I guess when I tweet and write in medium, those are what I believe in kind of a vasian way I'm not catching my claims that you would When you're writing a paper So I guess that's valid But I think a lot of people read into what I was saying More than what I was so when I say the algorithm Has a de-radicalizing influence. I'm just talking about the recommendations whereas a lot of people consider that to be Talking about all things considered so Even if it isn't doesn't have a bias towards a fringe maybe sociologically youtube Radicalizes people it could be the case. I don't know Um, but that's what i'm talking about. I'm talking about just the influence through recommendations And that's all we can hold google accountable for or at least it's what probably all could agree that google Should be held accountable for with its recommendation system Yeah, do you um, do you expect something to come or have you heard something to come out of youtube themselves? Like the the company any form of official statement to this? Nothing nothing at all. Um, the only I got a vague I got a vague a reporter was complaining that youtube sent them this So I think they've read it But I have no absolutely no contact with them Okay Cool, are you doing any anything in follow-up or do you have plans for more research? None of this i've just gone back to work i've applied a bunch for a bunch of independent grant money But i'm not optimistic. So if I don't get that i'll keep i'll keep it pattering along. I'll probably reduce the amount of recommendations because i'm spending like About 500 a month at the moment just keeping it running. So I gotta reduce my costs Yeah, and you do have a patreon for people to to chip into that, right? Yeah, so if you can link to that that'd be good. So if i'm getting something like Like 22 a month, so it doesn't really cover it Yeah all right, so Okay, this this has been very very pleasant. I think we've we've kind of looked at a lot of things is there anything you would like to amend To this that people should know about the research or about this this field No, I just have a um, I encourage you to have a play digging into data yourself. There's Um, if you're in this area the data is free to use the code's free to use Um, just consider this a contribution to knowledge Cool Well, thanks a lot mark. Um, I wish you a very pleasant evening for you, I guess and Cheers. Thanks Thanks for having me. Bye Bye
 Alright, I'm very pleased to have Mark Ladoitch here today
 in
 In he's the he's one of the authors of this paper. That's called algorithmic extremism examining YouTube's rabbit hole of radicalization
 So I've done a video about a topic like this before actually several and this is basically one in a line of
 research that examines the recommendation algorithm of YouTube specifically but also kind of the
 general
 Social media platforms. So Mark, thanks for being here
 Could you maybe for people who do not know anything about this could you kind of
 Explain where your work fits into what's been done before or kind of also what comes out of the of the mainstream
 Media about this topic because there's been quite a bit of of talk
 Yeah, so I'm not a researcher by trade I'm a programmer and the reason why I got into this was because I
 could see clear bias in the way
 The YouTube recommendation system is being reported on and also in the research
 a
 There's some narratives. I think it might be because
 There's a lot of people worried about rhyming populism and this is a way to explain that
 They're looking for ways YouTube are radicalizing people and finding evidence for that or
 But that could be anecdotes or in some of the studies at sexual
 quantitative data
 But they're only looking to confirm it. So there's really obvious things. I think you covered in your video
 Some of them will just look for movement towards alright channels
 Through like centrist
 Or alt-light they call it instead of looking for both ways. Just really obvious things like that
 Calling it calling it an infection that cliche clearly shows that really looked at it
 Like a curious person would so I thought I could easily as a software engineer just collect all the data
 And without any complicated statistics just looking at the overall
 Flow of recommendations between the two
 the overall flow of recommendations between videos
 What their political influences?
 Yeah, this this was a thing that that bugged me of the paper that I made a video about is that they claim there's this
 radicalization pipeline, right and with pipeline everyone sort of understands a different thing
 But I think the general consensus is that the recommendation algorithm itself will steer you
 towards like more extreme content and in this case towards like the
 alt-right extremist content and the paper actually
 analyzed and said
 Okay, we found evidence that there is movement in this direction
 But they never shown that this is significantly more movement than like in in the other direction
 So in order to justify a pipeline one would need to show that the movement this way is about larger than
 this way in some notion and
 So I've I've found I've actually spoken to the author of that paper and he agrees with that but
 Obviously doesn't
 Doesn't have like energy to go into every you know go
 Refute everything that comes at them. They've also been a bunch of like they've also been exposed to a lot of
 Criticism, let's say as have you and I think even more when when your paper came out, I think
 The four days there was just a giant storm of people
 Attacking your paper basically
 Basically just just listing every single thing that's wrong with it and why this isn't valid and
 Things like this. So let's actually jump into what you did specifically
 so
 if I'm if I'm can summarize and you can then maybe correct so that we can
 Establish what happened you basically collected?
 recommendations, so you scrape these videos on YouTube and you collected these recommendations and
 We can we can see this so in your paper you then can make such diagrams
 Such as this one or these
 where in the middle the white bar is a
 Channel or a group that you're interested in and then to the left you can see where all the
 impressions of that
 Channel or group come from so what's where where basically the views come from?
 Through the recommendation system and on the right you can see of all the views the channel has retrieved
 Where do they go to so what what what is recommended next?
 Right, so it basically shows both directions for for every group and then you've also labeled these by multiple
 methods so that you can kind of establish these groups and
 What is pretty cool?
 We've built this website where you can analyze this and my computer is a bit overloaded at the moment
 But I promise it's really interactive. All right, so during the interview my computer crashed
 So I'm doing this in post-production
 Just to show you how this website operates
 So what you have here is an overview over all the channels of what rare recommendations were collected
 And they are grouped into groups for example here after partisan left
 Center left social justice partisan right and so on so you can see for each group or channel where
 recommendations come from and where they go to
 For example the large red one here. I happen to know that is Fox News
 You can see Fox News of the daily impression it received from itself
 36 million impressions and it gives to itself 36 million these numbers have to agree by nature of how the data is collected
 of course
 But you can also see it gets 2.7 million impressions from CNN
 2.6 million from the next news network and so on and it also gives quite a bit of recommendations to CNN and so on so
 You can go for example at some individual channel. Here's the daily wire the daily wire is
 Mainly run by Ben Shapiro
 So it's a bit more to the right of Fox News and a bit more on the direction of alternative media
 You can see the daily wire gets some most of its impression
 Count wise from itself from the daily wire, but it gives most of them to Fox News
 So actually you can see here that itself is a long way down
 Like in whatever sixth or seventh place
 So actually if you were to watch the daily wire the recommendation system would most likely steer you towards something like Fox News
 Whereas the the claim is that the YouTube algorithm would actually steer you towards more radical content
 Actually in in reality, it seems like it would steer towards more of these mainstream content
 So actually want to go to this tab you can see different groupings here and
 The radicalization pathways is the previous paper we have looked at
 So they have all these channels here of this radicalization pathway and you can see here the control group
 gives very very very few
 Impressions to the IDW
 The IDW gives much more impressions to the control group, right?
 Again, and the IDW gives very few impressions to the alt light compared to the amount of
 Impressions the alt light gives to the IDW and even to the control group
 And if you look at the alt right and we're going to zoom in on that here
 It's even more so the alt right of course receives most of its impressions from itself
 Which you could expect for any kind of group. This is your classic filter bubble situation
 But if we analyze the question of is there a pipeline you can see that
 Next most likely you are diverted to the IDW and to the control group much more
 Than you come from the IDW or the control group, right?
 Let's look at the the alt light so called this is kind of the so called gateway to the control group
 So called gateway to the alt right you can see here the alt light gives most of its impressions next to itself
 To the control group and the IDW so deradicalizing
 If you look at its way to the alt right, you'll see that it gets about four times as much impressions
 From the alt right as it gives to the alt right. So
 Basically, it's kind of taking the steam out of a quarter of all of these sessions and gives it
 To either the control group or the IDW or itself
 So this is exactly the opposite of what you would expect
 If you were to claim that there is a pipeline
 You would expect their most recommendations to come from more moderate content and go towards more extreme content
 But it's exactly the opposite and again, these are the exact channels that this original paper used
 Now what this paper find that the one that we're discussing if you go to media type here
 What you'll be able to see is the division into mainstream media youtube creator and so-called missing link media
 Which we'll leave out for a moment
 Let's focus on mainstream versus youtube creators. You can see the mainstream media gives most recommendations to itself
 While giving only very little recommendations to youtube creators and the missing link media
 While the youtube creators actually give almost half of their impressions. Look at that
 They they like give almost half of their impressions to the mainstream media
 Which means that there is a big big push by the algorithm to
 Towards these mainstream media away from youtube creators. So in general and I invite you to look at this website
 In general, you can pretty much see that the exact opposite of a
 radicalization pipeline is happening if you of course if you look at these recommendations and how they are distributed actually
 Most recommendation pathways are towards moderate centrist content and of course creating creating filter bubbles
 Which is a problem by itself, but is not a radicalization pipeline
 Lastly, I want to look at white identitarians because it's a one of the groups that people are referring to when they
 Claim that there are these radicalization pipelines. Look at that
 So of the white identitarian they get most of their impressions, of course from other white identitarian
 Videos which would be your filter bubble phenomenon
 But they give most and this is a group right the white identitarian channels give most of their
 Recommendations to the partisan right to the central and left mass mainstream media
 libertarians and and so on and uh
 Themselves are like really really really far down
 So to claim that there is this radicalization pipeline if you look at this data to me
 Seems not justified from this data and if I look at the other paper
 That really left out the important analysis
 Of the the backwards direction
 It seems that given everything it seems that the claim is not warranted
 All right back to the interview. Um
 Is that about like what you've done is that a good summary of of the data collection and analysis
 Um, there's a yeah, it's a good summary I can go into detail. Yeah, please
 Um, so youtube doesn't make it easy so I started this back in november in 2018
 And I was using the youtube api
 And to get enough uh to get enough quota because they limit the amount of requests you can actually make to their api
 I created multiple keys, which is against their um policy
 Um, and they also asked you to delete all your data after 30 days
 That's also part of their policy. So um later
 about I think it was october
 2019 they cut off my access because I was doing that
 So I had to move to just uh scraping websites and now
 My collection process actually just loads up the website and gets the recommendations from the actual page like a user would
 Um
 And that's difficult because they block access after a couple of hundred requests. They'll
 They'll stop you that machine from actually requesting from the website
 So I need to
 Use a proxy service that
 That's fairly expensive and what they do is they simulate or they have actual residential connections through your home connection like atnt
 and
 my requests get tunneled through that like a variety of locations in the states to get um
 A representative kind of sample
 Cool so so the data collection is
 Would you say that's that's the hardest part?
 I feel the labeling of channels is also not so easy
 But you've you've managed to kind of do that
 Half automated also half collecting things from kind of um sources that analyze these channels
 But at least for for most of the things that i've inspected I found the labeling to be pretty sane
 I think this is always something you can attack the the original paper was also attacked on how they label
 I find this to be kind of vicarish
 Mostly I think your labels are pretty good as well. The other papers labels are also mostly pretty okay
 Yeah, so let's let's go to it. Sorry
 Yeah, it's quite subjective I expected the labeling to be what I get my pushback on but it turns out it was um
 the anonymous
 collection
 So what you've actually found here what are what would you say are your your main results and I can maybe
 Show
 So you've analyzed a bit where do things come from where do things go to and
 I found this this part here to be
 One of the even though it's pretty simple
 One of the core things to say about this is that mostly what you found
 could be
 said is
 It's simply a recommendation algorithm working as a recommendation algorithm should which means it creates
 You know your typical filter bubbles if you if I watch one minute of this video
 All of a sudden my site is filled with makeup tutorials and things like this
 But also you found that there is quite some
 Over the top push towards what could be considered mainstream media and there is
 A bit of a draw away from the smaller
 YouTuber like channels is that is that something that like is that character? I don't know
 That's right. So it yeah, that's a good way to characterize it if that chart we're looking at now
 If it was a neutral algorithm
 The green bars would be the same as the gray ones. So you you receive the same amount of recommendations as you give
 That would be proportional to the views that you get the future organically
 The recommendations that you get from the green bars
 That you get the future organically. Um the recommendations you receive be equivalent to that but we find that it disproportionately
 recommends mainstream media channels
 That's not even though. So it's not like um, it doesn't look like it's consistently doing that
 So you can find exceptions to that
 rule
 is um, I I believe one of the main criticisms of your paper has been that you
 Only use data from 2019 onwards
 and
 I have actually looked at your website and your website a lot of times says that the data has been collected from way earlier than that
 um, so is it that you've almost only used
 2019 data in your paper
 or what is in in the pipe the pipe is just from
 um november and december 2019
 and the reason we did that um
 Is that we only had 400 channels before that
 And the collection process have changed over time
 So this is a clean set of data we could look at and I thought the most recent was the most relevant
 So what is it doing now?
 But um, i've provided i've got the same analysis over time
 So i've got a gift that I made that everyone can look at which goes through all the months i've been collecting
 Um, and you can see that chart for where it goes to and has gone through a bunch of changes so in about april 2019
 That's when they really clamped down on conspiracies and other fringe channels
 Before that was it was much closer to neutral
 Okay, so but it never it never looked like a a rabbit hole it's never favoring
 Fringe channels. Yeah. I mean that that has been my experience also personally on youtube. I've
 I've joined youtube very early or i've i've watched youtube very early when
 Young earth creationism was still active and then these things were kind of completely discredited by simply
 having you having
 People exposed to other points of view and even I find this now
 Even though youtube makes it kind of easy to find let's say niche content
 It also exposes you to a bunch of of different views. Um, and and
 I've always found this to be very
 very
 optimistic in the sense of
 This is probably deradicalizing much more people than radicalizing
 But you've you've received like as I said a bunch of criticism in so if you could
 What was the
 The largest criticism irrespective of whether it was valid or not. What do you have you found was kind of what most people?
 were criticizing
 Most people criticizing that we were collecting anonymous recommendations. It wasn't the personalized ones
 Yeah, and it's actually like it is a valid limitation. We had it. There's a first limitation we talked about in this paper
 And
 It's still an open question
 How personalization would affect these aggregate results that we've got but I think it's reasonable
 To assume it will be quite similar once you average it out. So for any one person it might be different
 But you would expect personalization based on someone's history to even out because
 It's kind of the algorithms kind of like the average of all that when it's anonymous
 Yeah, I feel like you'll get that
 the
 the the notion of
 the the the notion that
 If because if you're not logged in the recommendation is like a person with only one video of history, right?
 So it's it's the same thing, but there's only one hit point of history instead of multiple I find
 Why should why should the behavior be qualitatively different if you have multiple points of history?
 like this is a strong claim that you have to you'd have to really show that there is a qualitative difference not just
 a more or less accuracy and I feel the people making this criticism are it's really on them to show that there is
 a substantial difference rather than saying that this is a giant limitation of the work
 Yeah, and it's also very hypocritical for a lot of the people saying it because
 some of them like
 Zion out who was mockingly saying that her article her original article in New York Times
 Used algo transparency, which is anonymous as well, but she doesn't she never looked into that
 I think a lot of this is completely motivated reasoning. They don't they don't care about the details
 I've I've seen this one this one twitter user
 She she comment she said something to the effect of if you've seen this article, please consult
 someone that works in this space like
 it's
 It's please don't don't read the article yourself. You must you must get your information through someone
 I've actually i've read the article
 I've I find it's pretty straightforward the limitations are clear
 But also the the results are pretty clear and it's it's actually mostly a boring article, right if if
 I'm sorry, like it's not a criticism. This is good. Like it's mostly you find that things work as expected
 There is a bit of a push towards mainstream which can be probably explaining that youtube wants to be
 Advertiser friendly right and these mainstream channels are already are
 Advertiser friendly so they probably get bumped a bit. Um, if what would you say is
 Maybe the most the most
 valid criticism that you've heard maybe not the biggest but the most
 Where do you where you say? Yeah, this is really
 This is really something that is you know
 I think um, I guess what's
 Um, there was criticism that i'm overclaiming not in the paper so much but in my tweets and medium
 I guess that's that's fair
 But I guess when I tweet and write in medium, those are what I believe in kind of a vasian way
 I'm not catching my claims that you would
 When you're writing a paper
 So I guess that's valid
 But I think a lot of people read into what I was saying
 More than what I was so when I say the algorithm
 Has a de-radicalizing influence. I'm just talking about the recommendations whereas a lot of people consider that to be
 Talking about all things considered so
 Even if it isn't doesn't have a bias towards a fringe maybe sociologically youtube
 Radicalizes people it could be the case. I don't know
 Um, but that's what i'm talking about. I'm talking about just the influence through recommendations
 And that's all we can hold google accountable for or at least it's what probably all could agree that google
 Should be held accountable for with its recommendation system
 Yeah, do you um, do you expect something to come or have you heard something to come out of youtube themselves?
 Like the the company any form of official statement to this?
 Nothing nothing at all. Um, the only I got a vague
 I got a vague a reporter was complaining that youtube sent them this
 So I think they've read it
 But I have no absolutely no contact with them
 Okay
 Cool, are you doing any anything in follow-up or do you have plans for more research?
 None of this i've just gone back to work i've applied a bunch for a bunch of independent grant money
 But i'm not optimistic. So if I don't get that i'll keep i'll keep it pattering along. I'll probably reduce the amount of recommendations
 because i'm spending like
 About 500 a month at the moment just keeping it running. So I gotta reduce my costs
 Yeah, and you do have a patreon for people to to chip into that, right?
 Yeah, so if you can link to that that'd be good. So if i'm getting something like
 Like 22 a month, so it doesn't really cover it
 Yeah
 all right, so
 Okay, this this has been very very pleasant. I think we've we've kind of looked at
 a lot of things is there anything you would like to
 amend
 To this that people should know about the research or about this this field
 No, I just have a um, I encourage you to have a play digging into data yourself. There's
 Um, if you're in this area the data is free to use the code's free to use
 Um, just consider this a contribution to knowledge
 Cool
 Well, thanks a lot mark. Um, I wish you a very pleasant evening for you, I guess
 and
 Cheers. Thanks
 Thanks for having me. Bye
 Bye