The Facebook “It’s Not Our Fault” Study

May 7th, 2015 by Christian

Today in Science, members of the Facebook data science team released a provocative study about adult Facebook users in the US “who volunteer their ideological affiliation in their profile.” The study “quantified the extent to which individuals encounter comparatively more or less diverse” hard news “while interacting via Facebook’s algorithmically ranked News Feed.”*

  • The research found that the user’s click rate on hard news is affected by the positioning of the content on the page by the filtering algorithm. The same link placed at the top of the feed is about 10-15% more likely to get a click than a link at position #40 (figure S5).
  • The Facebook news feed curation algorithm, “based on many factors,” removes hard news from diverse sources that you are less likely to agree with but it does not remove the hard news that you are likely to agree with (S7). They call news from a source you are less likely to agree with “cross-cutting.”*
  • The study then found that the algorithm filters out 1 in 20 cross-cutting hard news stories that a self-identified conservative sees (or 5%) and 1 in 13 cross-cutting hard news stories that a self-identified liberal sees (8%).
  • Finally, the research then showed that “individuals’ choices about what to consume” further limits their “exposure to cross-cutting content.” Conservatives will click on only 17% a little less than 30% of cross-cutting hard news, while liberals will click 7% a little more than 20% (figure 3).

My interpretation in three sentences:

  1. We would expect that people who are given the choice of what news they want to read will select sources they tend to agree with–more choice leads to more selectivity and polarization in news sources.
  2. Increasing political polarization is normatively a bad thing.
  3. Selectivity and polarization are happening on Facebook, and the news feed curation algorithm acts to modestly accelerate selectivity and polarization.

I think this should not be hugely surprising. For example, what else would a good filter algorithm be doing other than filtering for what it thinks you will like?

But what’s really provocative about this research is the unusual framing. This may go down in history as the “it’s not our fault” study.

 

Facebook: It’s not our fault.

I carefully wrote the above based on my interpretation of the results. Now that I’ve got that off my chest, let me tell you about how the Facebook data science team interprets these results. To start, my assumption was that news polarization is bad.  But the end of the Facebook study says:

“we do not pass judgment on the normative value of cross-cutting exposure”

This is strange, because there is a wide consensus that exposure to diverse news sources is foundational to democracy. Scholarly research about social media has–almost universally–expressed concern about the dangers of increasing selectivity and polarization. But it may be that you do not want to say that polarization is bad when you have just found that your own product increases it. (Modestly.)

And the sources cited just after this quote sure do say that exposure to diverse news sources is important. But the Facebook authors write:

“though normative scholars often argue that exposure to a diverse ‘marketplace of ideas’ is key to a healthy democracy (25), a number of studies find that exposure to cross-cutting viewpoints is associated with lower levels of political participation (22, 26, 27).”

So the authors present reduced exposure to diverse news as a “could be good, could be bad” but that’s just not fair. It’s just “bad.” There is no gang of political scientists arguing against exposure to diverse news sources.**

The Facebook study says it is important because:

“our work suggests that individuals are exposed to more cross-cutting discourse in social media they would be under the digital reality envisioned by some

Why so defensive? If you look at what is cited here, this quote is saying that this study showed that Facebook is better than a speculative dystopian future.*** Yet the people referred to by this word “some” didn’t provide any sort of point estimates that were meant to allow specific comparisons. On the subject of comparisons, the study goes on to say that:

“we conclusively establish that…individual choices more than algorithms limit exposure to attitude-challenging content.”

compared to algorithmic ranking, individuals’ choices about what to consume had a stronger effect”

Alarm bells are ringing for me. The tobacco industry might once have funded a study that says that smoking is less dangerous than coal mining, but here we have a study about coal miners smoking. Probably while they are in the coal mine. What I mean to say is that there is no scenario in which “user choices” vs. “the algorithm” can be traded off, because they happen together (Fig. 3 [top]). Users select from what the algorithm already filtered for them. It is a sequence.**** I think the proper statement about these two things is that they’re both bad — they both increase polarization and selectivity. As I said above, the algorithm appears to modestly increase the selectivity of users.

The only reason I can think of that the study is framed this way is as a kind of alibi. Facebook is saying: It’s not our fault! You do it too!

 

Are we the 4%?

In my summary at the top of this post, I wrote that the study was about people “who volunteer their ideological affiliation in their profile.” But the study also describes itself by saying:

“we utilize a large, comprehensive dataset from Facebook.”

“we examined how 10.1 million U.S. Facebook users interact”

These statements may be factually correct but I found them to be misleading. At first, I read this quickly and I took this to mean that out of the at least 200 million Americans who have used Facebook, the researchers selected a “large” sample that was representative of Facebook users, although this would not be representative of the US population. The “limitations” section discusses the demographics of “Facebook’s users,” as would be the normal thing to do if they were sampled. There is no information about the selection procedure in the article itself.

Instead, after reading down in the appendices, I realized that “comprehensive” refers to the survey research concept: “complete,” meaning that this was a non-probability, non-representative sample that included everyone on the Facebook platform. But out of hundreds of millions, we ended up with a study of 10.1m because users were excluded unless they met these four criteria:

  1. “18 or older”
  2. “log in at least 4/7 days per week”
  3. “have interacted with at least one link shared on Facebook that we classified as hard news”
  4. “self-report their ideological affiliation” in a way that was “interpretable”

That #4 is very significant. Who reports their ideological affiliation on their profile?

 

add your political views

 

It turns out that only 9% of Facebook users do that. Of those that report an affiliation, only 46% reported an affiliation in a way that was “interpretable.” That means this is a study about the 4% of Facebook users unusual enough to want to tell people their political affiliation on the profile page. That is a rare behavior.

More important than the frequency, though, is the fact that this selection procedure confounds the findings. We would expect that a small minority who publicly identifies an interpretable political orientation to be very likely to behave quite differently than the average person with respect to consuming ideological political news.  The research claims just don’t stand up against the selection procedure.

But the study is at pains to argue that (italics mine):

“we conclusively establish that on average in the context of Facebook, individual choices more than algorithms limit exposure to attitude-challenging content.”

The italicized portion is incorrect because the appendices explain that this is actually a study of a specific, unusual group of Facebook users. The study is designed in such a way that the selection for inclusion in the study is related to the results. (“Conclusively” therefore also feels out of place.)

Algorithmium: A Natural Element?

Last year there was a tremendous controversy about Facebook’s manipulation of the news feed for research. In the fracas it was revealed by one of the controversial study’s co-authors that based on the feedback received after the event, many people didn’t realize that the Facebook news feed was filtered at all. We also recently presented research with similar findings.

I mention this because when the study states it is about selection of content, who does the selection is important. There is no sense in this study that a user who chooses something is fundamentally different from the algorithm hiding something from them. While in fact the the filtering algorithm is driven by user choices (among other things), users don’t understand the relationship that their choices have to the outcome.

 

not sure if i hate facebook or everyone i know

 

In other words, the article’s strange comparison between “individual’s choices” and “the algorithm,” should be read as “things I choose to do” vs. the effect of “a process Facebook has designed without my knowledge or understanding.” Again, they can’t be compared in the way the article proposes because they aren’t equivalent.

I struggled with the framing of the article because the research talks about “the algorithm” as though it were an element of nature, or a naturally occurring process like convection or mitosis. There is also no sense that it changes over time or that it could be changed intentionally to support a different scenario.*****

Facebook is a private corporation with a terrible public relations problem. It is periodically rated one of the least popular companies in existence. It is currently facing serious government investigations into illegal practices in many countries, some of which stem from the manipulation of its news feed algorithm. In this context, I have to say that it doesn’t seem wise for these Facebook researchers to have spun these data so hard in this direction, which I would summarize as: the algorithm is less selective and less polarizing. Particularly when the research finding in their own study is actually that the Facebook algorithm is modestly more selective and more polarizing than living your life without it.

 

 

Update: (6pm Eastern)

Wow, if you think I was critical have a look at these. It turns out I am the moderate one.

Eszter Hargittai from Northwestern posted on Crooked Timber that we should “stop being mesmerized by large numbers and go back to taking the fundamentals of social science seriously.” And (my favorite): “I thought Science was a serious peer-reviewed publication.”

Nathan Jurgenson from Maryland and Snapchat wrote on Cyborgology (“in a fury“) that Facebook is intentionally “evading” its own role in the production of the news feed. “Facebook cannot take its own role in news seriously.” He accuses the authors of using the “Big-N trick” to intentionally distract from methodological shortcomings. He tweeted that “we need to discuss how very poor corporate big data research gets fast tracked into being published.”

Zeynep Tufekci from UNC wrote on Medium that “I cannot remember a worse apples to oranges comparison” and that the key take-away from the study is actually the ordering effects of the algorithm (which I did not address in this post). “Newsfeed placement is a profoundly powerful gatekeeper for click-through rates.”

 

Update: (5/10)

A comment helpfully pointed out that I used the wrong percentages in my fourth point when summarizing the piece. Fixed it, with changes marked.

 

Update: (5/15)

It’s now one week since the Science study. This post has now been cited/linked in The New York Times, Fortune, Time, Wired, Ars Technica, Fast Company, Engaget, and maybe even a few more. I am still getting emails. The conversation has fixated on the <4% sample, often saying something like: “So, Facebook said this was a study about cars, but it was actually only about blue cars.” That’s fine, but the other point in my post is about what is being claimed at all, no matter the sample.

I thought my “coal mine” metaphor about the algorithm would work but it has not always worked. So I’ve clamped my Webcam to my desk lamp and recorded a four-minute video to explain it again, this time with a drawing.******

Here’s the video:
https://www.youtube.com/watch?v=eBPntMSDGSs

If the coal mine metaphor failed me, what would be a better metaphor? I’m not sure. Suggestions?

 

 

 

Notes:

* Diversity in hard news, in their study, would be a self-identified liberal who receives a story from FoxNews.com, or a self-identified conservative who receives one from the HuffingtonPost.com, where the stories are about “national news, politics, [or] world affairs.” In more precise terms, for each user “cross-cutting content” was defined as stories that are more likely to be shared by partisans who do not have the same self-identified ideological affiliation that you do.

** I don’t want to make this even more nitpicky, so I’ll put this in a footnote. The paper’s citations to Mutz and Huckfeldt et al. to mean that “exposure to cross-cutting viewpoints is associated with lower levels of political participation” is just bizarre. I hope it is a typo. These authors don’t advocate against exposure to cross-cutting viewpoints.

*** Perhaps this could be a new Facebook motto used in advertising: “Facebook: Better than one speculative dystopian future!”

**** In fact, algorithm and user form a coupled system of at least two feedback loops. But that’s not helpful to measure “amount” in the way the study wants to, so I’ll just tuck it away down here.

***** Facebook is behind the algorithm but they are trying to peer-review research about it without disclosing how it works — which is a key part of the study. There is also no way to reproduce the research (or do a second study on a primary phenomenon under study, the algorithm) without access to the Facebook platform.

****** In this video, I intentionally conflate (1) the number of posts filtered and (2) the magnitude of the bias of the filtering. I did so because the difficulty with the comparison works the same way for both, and I was trying to make the example simpler. Thanks to Cedric Langbort for pointing out that “baseline error” is the clearest way of explaining this.

 

(This was cross-posted to The Social Media Collective and Wired.)

24 Responses to “The Facebook “It’s Not Our Fault” Study”

  1. You're Making Your Own Facebook Profile More Politically Biased Says:

    […] it’s your own decisions to click or ignore certain stories. However, some observers argue the Facebook study is flawed because of sampling problems and interpretation […]

  2. Facebook Study Says Users Make Their News Feed Less Diverse – TIME | The Artwork Science Says:

    […] one-sided, it’s your own decisions to click on or ignore certain stories. However, some observers argue the Facebook study is flawed because of sampling problems and interpretation […]

  3. Facebook Study Says Users Make Their News Feed Less Diverse – TIME | Everyday News Update Says:

    […] one-sided, it’s your own decisions to click on or ignore certain stories. However, some observers argue the Facebook study is flawed because of sampling problems and interpretation […]

  4. Free Wallpaper News | Facebook Study Says Users Make Their News Feed Less Diverse Says:

    […] one-sided, it’s your own decisions to click on or ignore certain stories. However, some observers argue the Facebook study is flawed because of sampling problems and interpretation […]

  5. The Facebook “It’s Not Our Fault” Study | Social Media Collective Says:

    […] (This post was cross-posted to multicast.) […]

  6. Facebook Study Says Users Make Their News Feed Less Diverse | I World New Says:

    […] one-sided, it’s your own decisions to click on or ignore certain stories. However, some observers argue the Facebook study is flawed because of sampling problems and interpretation […]

  7. Facebook Study Says Users Make Their News Feed Less Diverse. | paritoshpande95 Says:

    […] one-sided, it’s your own decisions to click on or ignore certain stories. However, some observers argue the Facebook study is flawed because of sampling problems and interpretation […]

  8. Facebook, filtering, polarization, and a flawed study? | Free the Blog Says:

    […] Christian Sandvig […]

  9. Facebook Study Says Users Make Their News Feed Less Diverse - Nfostation.com Says:

    […] one-sided, it’s your own decisions to click on or ignore certain stories. However, some observers argue the Facebook study is flawed because of sampling problems and interpretation […]

  10. Facebook Study Says Users Make Their News Feed Less DiverseBusiness Tech | Business Tech Says:

    […] it’s your own decisions to click on or ignore certain stories. However, some observers argue the Facebook study is flawed because of sampling problems and interpretation […]

  11. Facebook Study Says Users Make Their News Feed Less Diverse - itbusinessdirectitbusinessdirect Says:

    […] one-sided, it’s your own decisions to click on or ignore certain stories. However, some observers argue the Facebook study is flawed because of sampling problems and interpretation […]

  12. Facebook Dives Into the Echo Chamber | Science of News Says:

    […] but it’s a troubling model for studying months’ worth of Facebook user behavior. As Christian Sandvig points out, Facebook’s algorithm is based on what users engage with. In other words, if I tend to click […]

  13. Friday Futures: Cities, Cars, Spying on Cars, Drones that Spy on us all | The Fine Print Says:

    […] Research. Not so fast. Lots of problems with the study. Read the Fine Print, says Eszter Hargittai, Christian Sandvig, Zeynep Tufekci, Cyborgology, Eli Pariser (author of ‘The Filter […]

  14. Briefly | Stats Chat Says:

    […] You know that recent Facebook research saying it’s your fault your news feed is biased? Only 4% of US Facebook users provided enough political affiliation information to be included, and you have to read not just the research paper but the ‘Supplementary Materials’ to find this out (or read Eszter Hargittai). And, not surprisingly, the research doesn’t say the algorithms don’t increase bias: what else would a good filter algorithm be doing other than filtering for what it thinks you will … […]

  15. Facebook Study Says Users Make Their News Feed Less Diverse | AcrossTheFader.USAcrossTheFader.US Says:

    […] one-sided, it’s your own decisions to click on or ignore certain stories. However, some observers argue the Facebook study is flawed because of sampling problems and interpretation […]

  16. Facebook: Die Filterblase gibt es, aber wer ist schuld? - Netzpiloten.de Says:

    […] als jene, die unten gereiht werden. Forscher Christian Sandvig von der University of Michigan weist darauf hin, dass der News Feed bei Konservativen einen von 20, bei Liberalen einen von 13 konträren Artikeln […]

  17. Lo studio su Facebook, di Facebook Says:

    […] e interessi è più che altro colpa nostra. Alcuni hanno criticato la conclusione auto-assolutoria (The Facebook “It’s Not Our Fault” Study è il titolo di un articolo di Christian Sandvig sull’argomento). Zeynep Tufekci ha […]

  18. The Facebook “It’s Not Our Fault” Study | Social Media Collective | James Reads Says:

    […] was cross-posted to multicast and […]

  19. Facebook and the filter bubble | ARTS & FARCES internet Says:

    […] Christian Sandvig […]

  20. Facebook, qui nous influence le plus : l’algorithme ou nos a-priori ? | InternetActu Says:

    […] Pour Christian Sandvig, cette étude qui vise à disculper Facebook de toute responsabilité dans la polarisation de l’information (plusieurs chercheurs critiques appellent déjà cette étude “Ce n’est pas notre faute”), alors que l’algorithme semble avoir un effet, même s’il est moindre que les autres biais, est elle-même biaisée par la tentative de justification de ses auteurs, pointant par exemple le fait que l’exposition à des points de vue transversaux serait associée à des niveaux de participation politique moindre. Pour lui, cette étude est comparable à une étude que l’industrie du tabac pourrait commander pour montrer que le tabagisme est moins dangereux que les mines de charbon. Esther Hargittai, elle, pointe du doigt le problème de l’échantillonnage de l’étude, s’intéressant uniquement aux gens les plus politisés, des personnes qui ont une relation différente à la politique que l’utilisateur moyen. Autant de chercheurs qui demandent à Facebook d’être en tout cas plus clair sur ses buts quant à ses recherches sociologiques, souligne la Technology Review. Pour Gordon Hull : il n’y a pas de neutralité des Big Data, quoique tente de nous faire croire Facebook. Pour le sociologue Nathan Jurgenson : […]

  21. A culpa é dos algoritmos ou porque estamos nos odiando tanto - Scup Ideas Says:

    […] da Universidade de Harvard e Michigan, Christian Sandvig, concedeu ao estudo o título fantasia de “The Facebook ‘It’s Not Our Fault’ Study”, pois isenta as redes sociais de culpa no processo de alienação. O pesquisador afirma ainda que […]

  22. Aggregation #1 | The Press of Jess Says:

    […] put up. There is something to be said about the placing of context on a Facebook page as well.  Studies show that the users click rate on hard news is affected by how and where the link is positions on […]

  23. rickschmidt Says:

    Thanks. Great article.

  24. If you use Facebook to get your news, please — for the love of democracy … - Nfostation.com Says:

    […] of information in the News Feed. The communications scholar Christian Sandvig called it the “‘it’s not our fault’ study” — a peculiarly deliberate attempt to prove that, even if filter bubbles do exist, their algorithm […]

Leave a Reply

Bad Behavior has blocked 168 access attempts in the last 7 days.