# Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists

Originally published: 2019-09-23
Canonical URL: /2019/Sep/heads-i-win-tails-never-heard-of-her-or-selective-reporting-and-the-tragedy-of-the-green-rationalists/

[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting)

**Followup to:** [What Evidence Filtered Evidence?](https://www.lesswrong.com/posts/kJiPnaQPiy4p9Eqki/what-evidence-filtered-evidence) 

In ["What Evidence Filtered Evidence?"](https://www.lesswrong.com/posts/kJiPnaQPiy4p9Eqki/what-evidence-filtered-evidence), we are asked to consider a scenario involving a coin that is _either_ biased to land Heads 2/3rds of the time, _or_ Tails 2/3rds of the time. Observing Heads is 1 bit of evidence for the coin being Heads-biased (because the Heads-biased coin lands Heads with probability 2/3, the Tails-biased coin does so with probability 1/3, the likelihood ratio of these is $\frac{2/3}{1/3} = 2$, and $\log_{2} 2 = 1$), and analogously and respectively for Tails.

If such a coin is flipped ten times by someone who [doesn't make literally false statements](https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases#2__The_law_of_no_literal_falsehood_), who then reports that the 4th, 6th, and 9th flips came up Heads, then the update to our beliefs about the coin depends on what _algorithm_ the not-lying[^honest] reporter used to decide to report those flips in particular. If they always report the 4th, 6th, and 9th flips _independently_ of the flip outcomes—if there's no [evidential entanglement](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence) between the flip outcomes and the choice of which flips get reported—then reported flip-outcomes can be treated the same as flips you observed yourself: three Headses is 3 * 1 = 3 bits of evidence in favor of the hypothesis that the coin is Heads-biased. (So if we were initially 50:50 on the question of which way the coin is biased, our posterior odds after collecting 3 bits of evidence for a Heads-biased coin would be $2^3:1$ = 8:1, or a probability of 8/(1 + 8) ≈ 0.89 that the coin is Heads-biased.)

[^honest]: I don't quite want to use the word _honest_ here.

On the other hand, if the reporter mentions only and exactly the flips that came out Heads, then we can _infer_ that the other 7 flips came out Tails (if they didn't, the reporter would have mentioned them), giving us posterior odds of $2^3:2^7$ = 1:16, or a probability of around 0.06 that the coin is Heads-biased.

So far, so standard. (You _did_ [read the Sequences](https://www.readthesequences.com/), right??) What I'd like to _emphasize_ about this scenario today, however, is that while a Bayesian reasoner who _knows_ the non-lying reporter's algorithm of what flips to report will never be misled by the selective reporting of flips, a Bayesian with _mistaken_ beliefs about the reporter's decision algorithm can be misled _quite badly_: compare the 0.89 and 0.06 probabilities we just derived given the _same_ reported outcomes, but different assumptions about the reporting algorithm.

If the coin gets flipped a sufficiently large number of times, a reporter whom you _trust_ to be impartial (but isn't), can _make you believe anything she wants without ever telling a single lie_, just with [appropriate selective reporting](https://slatestarcodex.com/2015/09/16/cardiologists-and-chinese-robbers/). Imagine a _very_ biased coin that comes up Heads 99% of the time. If it gets flipped ten thousand times, 100 of those flips will be Tails (in expectation), giving a selective reporter plenty of examples to point to if she wants to convince you that the coin is extremely Tails-biased.

Toy models about biased coins are instructive for constructing examples with explicitly calculable probabilities, but the same _structure_ applies to any real-world situation where you're receiving evidence from other agents, and you have uncertainty about what algorithm is being used to determine what reports get to you. Reality is like the coin's bias; evidence and arguments are like the outcome of a particular flip. _Wrong_ theories [will still have _some_ valid arguments and evidence supporting them](https://www.lesswrong.com/posts/627DZcvme7nLDrbZu/update-yourself-incrementally) (as even a very Heads-biased coin will come up Tails sometimes), but theories that are [_less_ wrong](https://tvtropes.org/pmwiki/pmwiki.php/Main/TitleDrop) will have _more_.

If selective reporting is mostly due to the idiosyncratic [bad intent](http://benjaminrosshoffman.com/bad-faith-behavior-not-feeling/) of rare malicious actors, then you might hope for safety in [(the law of large)](https://en.wikipedia.org/wiki/Law_of_large_numbers) numbers: if Helga in particular is systematically more likely to report Headses than Tailses that she sees, then her flip reports will diverge from everyone else's, and you can take that into account when reading Helga's reports. On the other hand, if selective reporting is mostly due to systemic _structural_ factors that result in _correlated_ selective reporting even among well-intentioned people who are being honest as best they know how,[^how] then you might have a more serious problem.

[^how]: And it turns out that knowing _how_ to be honest is _much more work_ than one might initially think. You _have_ [read the Sequences](https://www.readthesequences.com/), right?!

["A Fable of Science and Politics"](https://www.lesswrong.com/posts/6hfGNLf4Hg5DXqJCF/a-fable-of-science-and-politics) depicts a fictional underground Society polarized between two partisan factions, the Blues and the Greens. "[T]here is a 'Blue' and a 'Green' position on almost every contemporary issue of political or cultural importance." If human brains consistently understood [the is/ought distinction](https://plato.stanford.edu/entries/hume-moral/#io), then political or cultural alignment with the Blue or Green agenda wouldn't distort people's beliefs about reality. Unfortunately ... humans. (I'm not even going to finish the sentence.)

Reality itself isn't on anyone's side, but any particular fact, argument, [sign, or portent](https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-line) might just so happen to be more easily construed as "supporting" the Blues or the Greens. The Blues want stronger marriage laws; the Greens want no-fault divorce. An [evolutionary psychologist](https://www.lesswrong.com/posts/epZLSoNvjW53tqNj9/evolutionary-psychology) investigating [effects of kin-recognition mechanisms on child abuse by stepparents](https://en.wikipedia.org/wiki/Cinderella_effect) might aspire to scientific objectivity, but being objective and _staying_ objective is _difficult_ when you're embedded in an [intelligent social web](https://www.lesswrong.com/posts/AqbWna2S85pFTsHH4/the-intelligent-social-web) in which in your work is going to be predictably championed by Blues and reviled by Greens.

Let's make another toy model to try to understand the resulting distortions on the Undergrounders' collective epistemology. Suppose Reality is a coin—no, not a coin, a three-sided die,[^triangle] with faces colored blue, green, and gray. One-third of the time it comes up blue (representing a fact that is more easily construed as supporting the Blue narrative), one-third of the time it comes up green (representing a fact that is more easily construed as supporting the Green narrative), and one-third of the time it comes up gray (representing a fact that not even the worst ideologues know how to spin as "supporting" their side). 

[^triangle]: For lack of an appropriate [Platonic solid](https://en.wikipedia.org/wiki/Platonic_solid) in three-dimensional space, maybe imagine tossing a triangle in two-dimensional space??

Suppose each faction has social-punishment mechanisms enforcing consensus internally. [Without loss of generality](https://en.wikipedia.org/wiki/Without_loss_of_generality), take the Greens (with the understanding that everything that follows goes just the same if you swap "Green" for "Blue" and _vice versa_).[^choice] People observe rolls of the die of Reality, and can freely choose what rolls to report—except a resident of a Green city who reports more than 1 blue roll for every 3 green rolls is assumed to be a secret Blue Bad Guy, and faces increasing social punishment as their ratio of reported green to blue rolls falls below 3:1. (Reporting gray rolls is always safe.)

[^choice]: As an author, I'm facing some conflicting desiderata in my color choices here. I want to say "Blues and Greens" _in that order_ for consistency with "A Fable of Science and Politics" (and other [classics from the Sequences](https://www.lesswrong.com/posts/uaPc4NHi5jGXGQKFS/blue-or-green-on-regulation)). Then when making an arbitrary choice to talk in terms of one of the factions in order to avoid cluttering the exposition, you might have expected me to say "Without loss of generality, take the Blues," because the _first_ item in a sequence ("Blues" in "Blues and Greens") is a more of a [Schelling point](https://www.lesswrong.com/posts/yJfBzcDL9fBHJfZ6P/nash-equilibria-and-schelling-points) than the second, or last, item. But I don't _want_ to take the Blues, because that color choice [has other associations](http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/) that I'm trying to avoid right now: if I said "take the Blues", I fear many readers would assume that I'm trying to directly push a partisan point about [soft censorship](https://slatestarcodex.com/2019/04/02/social-censorship-the-first-offender-model/) and [preference-falsification](https://en.wikipedia.org/wiki/Preference_falsification) social pressures in liberal/left-leaning subcultures in the contemporary United States. To be fair, it's _true_ that soft censorship and preference-falsification social pressures in liberal/left-leaning subcultures in the contemporary United States are, historically, what inspired me, personally, to write this post. It's okay for you to notice that! But I'm _trying_ to talk about the _general mechanisms_ that generate this _class_ of distortions on a Society's collective epistemology, independently of which faction or which ideology happens to be "on top" [in a particular place and time](https://www.lesswrong.com/posts/cKrgy7hLdszkse2pq/archimedes-s-chronophone). If I'm _doing my job right_, then my analogue in a ["nearby" Everett branch](https://www.lesswrong.com/posts/WqGCaRhib42dhKWRL/if-many-worlds-had-come-first) whose local subculture was as "right-polarized" as my Berkeley environment is "left-polarized", would have written a post making the _same_ arguments.

The punishment is typically _informal_: there's no _official_ censorship from Green-controlled local governments, just a visible incentive gradient made out of social-media pile-ons, denied promotions, lost friends and mating opportunities, increased risk of being involuntarily committed to psychiatric prison,[^prison] _&c._ Even people who privately agree with dissident speech might participate in punishing it, the better to evade punishment themselves.

[^prison]: Okay, they market themselves as psychiatric "hospitals", but let's not be confused by [misleading labels](https://www.lesswrong.com/posts/ZXuqNhMDcs6mYtb6i/the-american-system-and-misleading-labels).

This scenario presents a problem for people who live in Green cities who want to make _and share_ accurate models of reality. It's impossible to report _every_ die roll (the only _1:1 scale_ map of the territory, is the territory itself), but it seems clear that the most generally useful models—the ones you would expect arbitrary AIs to come up with—aren't going to be sensitive to which facts are "blue" or "green". The reports of aspiring epistemic rationalists who are _just trying to make sense of the world_ will end up being about one-third blue, one-third green, and one-third gray, matching the distribution of the Reality die.

From the perspective of ordinary nice smart Green citizens who have not been trained in [the Way](http://yudkowsky.net/rational/virtues/), these reports look _unthinkably_ Blue. Aspiring epistemic rationalists who are actually [paying attention](https://srconstantin.wordpress.com/2019/02/25/humans-who-are-not-concentrating-are-not-general-intelligences/) can easily distinguish Blue partisans from actual truthseekers,[^bias] but the [social-punishment machinery](http://benjaminrosshoffman.com/blame-games/) can't process more than [five words at a time](https://www.lesswrong.com/posts/4ZvJab25tDebB8FGE/you-have-about-five-words). The social consequences of being an _actual_ Blue Bad Guy, or just an honest nerd who doesn't know when to keep her stupid trap shut, are the same.

[^bias]: Or rather, aspiring epistemic rationalists can do a _decent_ job of assessing the _extent to which_ someone is exhibiting truth-tracking behavior, or Blue-partisan behavior. Obviously, people who are _consciously_ trying to seek truth, are not necessarily going to _succeed_ at [overcoming bias](https://tvtropes.org/pmwiki/pmwiki.php/Main/TitleDrop), and attempts to correct for the "pro-Green" distortionary forces being discussed in this parable could easily veer into "pro-Blue" _over_-correction.

In this scenario,[^assumptions] public opinion within a subculture or community in a Green area is constrained by the 3:1 (green:blue) "[Overton](https://en.wikipedia.org/wiki/Overton_window) ratio." In particular, under these conditions, it's _impossible to have a rationalist community_—at least the most naïve conception of such. If your marketing literature _says_, "Speak the truth, even if your voice trembles," but all the [savvy](https://slatestarcodex.com/2017/10/23/kolmogorov-complicity-and-the-parable-of-lightning/) high-status people's actual _reporting algorithm_ is, "Speak the truth, except when that would cause the local social-punishment machinery to mark me as a Blue Bad Guy and hurt me and any people or institutions I'm associated with—in which case, tell the most convenient lie-of-omission", then [smart sincere](https://www.overcomingbias.com/2009/12/the-smart-sincere-syndrome.html) idealists who have internalized your marketing literature as a moral ideal and trust the community to implement that ideal, are going to be _misled_ by the community's stated beliefs—and _confused_ at some of the pushback they get when submitting reports with a 1:1:1 blue:green:gray ratio.

[^assumptions]: Please be appropriately skeptical about the real-world relevance of my made-up modeling assumptions! If it turned out that my choice of assumptions were (subconsciously) selected for the resulting conclusions about how bad evidence-filtering is, that would be really bad for the same reason that I'm claiming that evidence-filtering is really bad!

Well, misled to _some_ extent—maybe not much! In the absence of an [Oracle AI](https://wiki.lesswrong.com/wiki/Oracle_AI) (or a competing rationalist community in Blue territory) to compare notes with, then it's not clear how one could get a _better_ map than trusting what the "green rationalists" say. With a few more made-up modeling assumptions, we can _quantify_ the distortion introduced by the Overton-ratio constraint, which will hopefully help develop an _intuition_ for how large of a problem this sort of thing might be in real life.

Imagine that Society needs to make a decision about an Issue (like a question about divorce law or merchant taxes). Suppose that the facts relevant to making optimal decisions about an Issue are represented by nine rolls of the Reality die, and that the quality (utility) of Society's decision is proportional to the (base-two logarithm) entropy of the distribution of what facts get heard and discussed.[^entropy]

[^entropy]: The entropy of a discrete probability distribution is maximized by the uniform distribution, in which all outcomes receive equal probability-mass. I only chose these "exactly nine equally-relevant facts/rolls" and "entropic utility" assumptions to make the arithmetic easy on me; a more realistic model might admit arbitrarily many facts into discussion of the Issue, but posit a distribution of facts/rolls with [diminishing marginal](https://en.wikipedia.org/wiki/Marginal_utility#Diminishing_marginal_utility) relevance to Society's decision quality.

The maximum achievable decision quality is $\log_{2} 9$ ≈ 3.17.

On average, Green partisans will find 3 "green" facts[^scare] and 3 "gray" facts to report, and mercilessly [stonewall](https://www.lesswrong.com/posts/wqmmv6NraYv4Xoeyj/conversation-halters) anyone who tries to report any "blue" facts, for a decision quality of $\log_{2} 6$ ≈ 2.58.

[^scare]: The scare quotes around the adjective "'green'" (_&c._) when applied to the word "fact" (as opposed to a die roll outcome _representing_ a fact in our toy model) are significant! The facts aren't actually on anyone's side! We're trying to model the _distortions_ that arise from stupid humans _thinking_ that the facts are on someone's side! This is sufficiently important—and difficult to remember—that I should probably repeat it until it becomes obnoxious!

On average, the Overton-constrained rationalists will report the same 3 "green" and 3 "gray" facts, but something interesting happens with "blue" facts: each individual can only afford to report one "blue" fact without blowing their Overton budget—but it doesn't have to be the _same_ fact for each person. Reports of all 3 (on average) blue rolls get to enter the public discussion, but get mentioned (cited, retweeted, _&c_.) 1/3 as often as green or gray rolls, in accordance with the Overton ratio. So it turns out that the constrained rationalists end up with a decision quality of $\frac{6}{7} \log_{2} 7 + \frac{1}{7} \log_{2} 21$ ≈ 3.03,[^calculation] _significantly_ better than the Green partisans—but still falling short of the theoretical ideal where all the relevant facts get their due attention.

[^calculation]: You have three green slots, three gray slots, and three blue slots. You put three counters each on each of the green and gray slots, and one counter each on each of the blue slots. The frequencies of counters per slot is [3, 3, 3, 3, 3, 3, 1, 1, 1]. The total number of counters you put down is 3*6 + 3 = 18 + 3 = 21. To turn the frequencies into a probability distribution, you divide everything by 21, to get [1/7, 1/7, 1/7, 1/7, 1/7, 1/7, 1/21, 1/21, 1/21]. Then the entropy is $6\cdot-\frac{1}{7}\log_{2}\frac{1}{7}+3\cdot-\frac{1}{21}\log_{2}\frac{1}{21}$, which simplifies to $\frac{6}{7}\log_{2}7+\frac{1}{7}\log_{2}21$.

If it's just not _pragmatic_ to expect people to defy [their incentives](https://www.lesswrong.com/posts/5nH5Qtax9ae8CQjZ9/no-it-s-not-the-incentives-it-s-you), is this the best we can do? Accept a somewhat distorted state of discourse, forever?

At least one _partial_ remedy seems apparent. Recall from our original coin-flipping example that a Bayesian who _knows_ what the filtering process looks like, can take it into account and make the correct update. If you're filtering your evidence to avoid social punishment, but it's possible to clue in your fellow rationalists to your _filtering algorithm_ without triggering the social-punishment machinery—you mustn't [assume that everyone already knows](https://www.lesswrong.com/posts/BNfL58ijGawgpkh9b/everybody-knows)!—that's potentially a big win. In other words, [blatant cherry-picking is the best kind!](https://www.lesswrong.com/posts/KzAG4yWQJosmEjHe2/blatant-lies-are-the-best-kind)