From 881b8c7cff4fe4f1c08d9586af5010d0786c8fce Mon Sep 17 00:00:00 2001
From: "Zack M. Davis" <code@zackmdavis.net>
Date: Wed, 15 Jul 2026 10:52:41 -0700
Subject: [PATCH] import Less Wrong exclusives

I was imagining doing this manually, but delegating to Claude Code and
spot-checking is so much more efficient
---
 content/2019/algorithms-of-deception.md       | 157 +++++++-
 ...esnt-mean-youre-stupid-and-bad-probably.md |  37 +-
 content/2019/but-it-doesnt-matter.md          |  10 +
 ...y-useful-than-one-might-initially-think.md | 107 +++++
 ...d-the-tragedy-of-the-green-rationalists.md |  74 +++-
 content/2019/maybe-lying-doesnt-exist.md      |  99 ++++-
 ...s-the-decoupling-contextualizing-binary.md |  40 +-
 ...-categories-and-simple-membership-tests.md |  97 ++++-
 ...-and-dishonesty-explain-each-other-away.md |  28 ++
 content/2019/the-univariate-fallacy.md        | 104 ++++-
 content/2019/where-to-draw-the-boundaries.md  | 157 +++++++-
 ...onian-generalized-anti-zombie-principle.md | 100 +++++
 .../blogging-on-less-wrong-2020-upper-half.md |  13 -
 ...n-endogenous-epistemic-factionalization.md | 152 ++++++++
 .../december-2019-blogging-on-less-wrong.md   |  13 -
 .../dont-double-crux-with-suicide-rock.md     |  28 ++
 content/2020/maybe-lying-cant-exist.md        |  64 +++
 content/2020/message-length.md                | 220 +++++++++++
 content/2020/msg-len.md                       |  15 +
 ...g-lay-theories-through-graphical-models.md |  56 +++
 ...line-basics-of-the-evolution-of-meaning.md | 298 ++++++++++++++
 ...inc-vs-the-efficient-markets-hypothesis.md |  28 ++
 .../blogging-on-less-wrong-2020-lower-half.md |  11 -
 content/2021/blood-is-thicker-than-water.md   | 112 ++++++
 .../comment-on-deception-as-cooperation.md    |  52 +++
 ...-interests-or-differential-signal-costs.md |  28 ++
 content/2021/feature-selection.md             | 309 +++++++++++++++
 .../2021/reply-to-nate-soares-on-dolphins.md  |  69 ++++
 ...-categories-are-optimized-for-deception.md | 366 ++++++++++++++++++
 content/2022/blogging-on-less-wrong-2021.md   |  12 -
 ...ns-concerning-digital-minds-and-society.md |  76 ++++
 ...onvergence-is-like-discouraging-betting.md | 124 ++++++
 ...ns-of-llm-successes-a-debate-in-one-act.md | 166 ++++++++
 content/2023/assume-bad-faith.md              |  70 ++++
 ...esian-networks-arent-necessarily-causal.md | 102 +++++
 .../conflict-theory-of-bounded-distrust.md    |  36 ++
 .../is-there-anything-thats-worth-more.md     |  26 ++
 content/2023/justice-cherryl.md               | 122 ++++++
 ...-of-social-grace-is-an-epistemic-virtue.md |  88 +++++
 ...list-discourse-is-like-physicist-motors.md |  86 ++++
 .../youll-never-persuade-people-like-that.md  |  36 ++
 .../and-all-the-shoggoths-merely-players.md   | 146 +++++++
 .../2024/comment-on-death-and-the-gorgon.md   |  76 ++++
 content/2024/ironing-out-the-squiggles.md     | 104 +++++
 ...lmanning-is-normal-itt-passing-is-niche.md |  52 +++
 ...umans-was-net-negative-for-human-values.md |  26 ++
 content/2024/the-standard-analogy.md          | 140 +++++++
 ...our-layers-of-intellectual-conversation.md |  48 +++
 ...-contributions-are-logically-irrelevant.md |  68 ++++
 content/2025/just-make-a-new-rule.md          |  38 ++
 50 files changed, 4406 insertions(+), 80 deletions(-)
 create mode 100644 content/2019/but-it-doesnt-matter.md
 create mode 100644 content/2019/firming-up-not-lying-around-its-edge-cases-is-less-broadly-useful-than-one-might-initially-think.md
 create mode 100644 content/2019/stupidity-and-dishonesty-explain-each-other-away.md
 create mode 100644 content/2020/algorithmic-intent-a-hansonian-generalized-anti-zombie-principle.md
 delete mode 100644 content/2020/blogging-on-less-wrong-2020-upper-half.md
 create mode 100644 content/2020/comment-on-endogenous-epistemic-factionalization.md
 delete mode 100644 content/2020/december-2019-blogging-on-less-wrong.md
 create mode 100644 content/2020/dont-double-crux-with-suicide-rock.md
 create mode 100644 content/2020/maybe-lying-cant-exist.md
 create mode 100644 content/2020/message-length.md
 create mode 100644 content/2020/msg-len.md
 create mode 100644 content/2020/optimized-propaganda-with-bayesian-networks-comment-on-articulating-lay-theories-through-graphical-models.md
 create mode 100644 content/2020/philosophy-in-the-darkest-timeline-basics-of-the-evolution-of-meaning.md
 create mode 100644 content/2020/zoom-technologies-inc-vs-the-efficient-markets-hypothesis.md
 delete mode 100644 content/2021/blogging-on-less-wrong-2020-lower-half.md
 create mode 100644 content/2021/blood-is-thicker-than-water.md
 create mode 100644 content/2021/comment-on-deception-as-cooperation.md
 create mode 100644 content/2021/communication-requires-common-interests-or-differential-signal-costs.md
 create mode 100644 content/2021/feature-selection.md
 create mode 100644 content/2021/reply-to-nate-soares-on-dolphins.md
 create mode 100644 content/2021/unnatural-categories-are-optimized-for-deception.md
 delete mode 100644 content/2022/blogging-on-less-wrong-2021.md
 create mode 100644 content/2022/comment-on-propositions-concerning-digital-minds-and-society.md
 create mode 100644 content/2023/aiming-for-convergence-is-like-discouraging-betting.md
 create mode 100644 content/2023/alignment-implications-of-llm-successes-a-debate-in-one-act.md
 create mode 100644 content/2023/assume-bad-faith.md
 create mode 100644 content/2023/bayesian-networks-arent-necessarily-causal.md
 create mode 100644 content/2023/conflict-theory-of-bounded-distrust.md
 create mode 100644 content/2023/is-there-anything-thats-worth-more.md
 create mode 100644 content/2023/justice-cherryl.md
 create mode 100644 content/2023/lack-of-social-grace-is-an-epistemic-virtue.md
 create mode 100644 content/2023/rationalist-discourse-is-like-physicist-motors.md
 create mode 100644 content/2023/youll-never-persuade-people-like-that.md
 create mode 100644 content/2024/and-all-the-shoggoths-merely-players.md
 create mode 100644 content/2024/comment-on-death-and-the-gorgon.md
 create mode 100644 content/2024/ironing-out-the-squiggles.md
 create mode 100644 content/2024/on-the-contrary-steelmanning-is-normal-itt-passing-is-niche.md
 create mode 100644 content/2024/the-evolution-of-humans-was-net-negative-for-human-values.md
 create mode 100644 content/2024/the-standard-analogy.md
 create mode 100644 content/2025/comment-on-four-layers-of-intellectual-conversation.md
 create mode 100644 content/2025/critic-contributions-are-logically-irrelevant.md
 create mode 100644 content/2025/just-make-a-new-rule.md

diff --git a/content/2019/algorithms-of-deception.md b/content/2019/algorithms-of-deception.md
index 48d00a4..905fc16 100644
--- a/content/2019/algorithms-of-deception.md
+++ b/content/2019/algorithms-of-deception.md
@@ -1,8 +1,159 @@
-Title: "Algorithms of Deception!"
+Title: Algorithms of Deception!
 Date: 2019-10-19 11:06
 Status: published
 Category: philosophy
-Tags: elsewhere
+Tags: rationality, honesty, Python
 Slug: algorithms-of-deception
 
-[(new post on _Less Wrong_)](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception)
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception)
+
+I want you to imagine a world consisting of a sequence of independent and identically distributed random variables $X_i$, and two computer programs.
+
+The first program is called Reporter. As input, it accepts a bunch of the random variables $X_i$. As output, it returns a list of sets whose elements belong to the domain of the $X_i$.
+
+The second program is called Audience. As input, it accepts the output of Reporter. As output, it returns a probability distribution.
+
+Suppose the $X_i$ are drawn from the following distribution:
+
+$$P(X = x) = \begin{cases} 1/2 & x = 1 \\ 1/4 & x = 2 \\ 3/16 & x = 3 \\ 1/16 & x = 4 \\ \end{cases}$$
+
+We can model drawing a sample from this distribution using this function in the [Python programming language](https://www.python.org/):
+
+```python
+import random
+
+def x():
+    r = random.random()
+    if 0 <= r < 1/2:
+        return 1
+    elif 1/2 <= r < 3/4:
+        return 2
+    elif 3/4 <= r < 15/16:
+        return 3
+    else:
+        return 4
+```
+
+For compatibility, we can imagine that Reporter and Audience are also written in Python. This is just for demonstration in the blog post that I'm writing—the _real_ Reporter and Audience (out there in the world I'm asking you to imagine) might be much more complicated programs written for some kind of _alien_ computer the likes of which we have not yet dreamt! But I like Python, and for the moment, we can pretend.
+
+So pretend that Audience looks like this (where the dictionary, or hashmap, that gets returned represents a probability distribution, with the keys being random-variable outcomes and the values being probabilities):
+
+```python
+from collections import Counter
+
+def audience(report):
+    a = Counter()
+    for sight in report:
+        for possibility in sight:
+            a[possibility] += 1/len(sight)            
+    d = sum(a_j - len(a) for a_j in a.values())
+    return {x: (a_i - 1)/d for x, a_i in a.items()}
+```
+
+Let's consider multiple possibilities for the form that Reporter could take. A particularly simple implementation of Reporter (call it `reporter_0`) might look like this:
+
+```python
+def reporter_0(xs):
+    output = []
+    for x in xs:
+        output.append({x})
+    return output
+```
+
+The pairing of `audience` and `reporter_0` has a _Very Interesting Property!_ When we call our Audience on the output of this Reporter, the probability distribution that Audience returns is _very similar_ to the distribution that our random variables are from![^wrong]
+
+[^wrong]: But _only_ "very" similar: the code for `audience` is _not_ the mathematically correct thing to do in this situation; it's just an approximation that ought to be good enough for the point I'm trying to make in this blog post, for which I'm trying to keep the code simple. (Specifically, the last two lines of `audience` are based on [the mode of the Dirichlet distribution](https://en.wikipedia.org/wiki/Dirichlet_distribution#Mode), but, firstly, that part about increasing the hyperparameters fractionally when you're uncertain about what was observed (`a[possibility] += 1/len(sight)`) is pretty dodgy, and secondly, if you were _actually_ going to try to predict an outcome drawn from a [categorical distribution](https://en.wikipedia.org/wiki/Categorical_distribution) like $P(X)$ using the Dirichlet distribution as a [conjugate prior](https://en.wikipedia.org/wiki/Conjugate_prior), you'd need to integrate over the Dirichlet hyperparameters; you shouldn't just pretend that the mode/peak represents the true parameters of the categorical distribution—but as I said, we _are_ just pretending.)
+
+```
+>>> audience(reporter_0([x() for _ in range(100000)]))
+{1: 0.5003300528084493, 2: 0.2502900464074252, 3: 0.1873799807969275, 4: 0.062119939190270444}
+
+# Compare to P(X) expressed as a Python dictionary—
+>>> {1: 1/2, 2: 1/4, 3: 3/16, 4: 1/16}
+{1: 0.5, 2: 0.25, 3: 0.1875, 4: 0.0625}
+```
+
+Weird, right?!
+
+Of course, there are _other_ possible implementations of Reporter. For example, this choice of Reporter (`reporter_1`) does _not_ result in the Very Interesting Property—
+
+```python
+def reporter_1(xs):
+    output = []
+    for _ in range(len(xs)):
+        output.append({4})
+    return output
+```
+
+It instead induces Audience to output a very different (and rather boring) distribution. It doesn't even matter how the $X_i$ turned up; the result will always be the same:
+
+```
+>>> audience(reporter_1([x() for _ in range(100000)]))
+{4: 1.0}
+```
+
+We could go on imagining other versions of Reporter, like this one (`reporter_2`)—
+
+```
+def reporter_2(xs):
+    output = []
+    for x in xs:
+        if x == 4 or random.random() < 0.2:
+            output.append({x})
+        else:
+            continue
+    return output
+```
+
+While the distribution that `reporter_2` makes Audience output isn't as boring as the one we saw for `reporter_1`, it still doesn't result in the Very Interesting Property of matching the distribution of the $X_i$. It comes _closer_ than `reporter_1` did—notice how the _ratios_ of probabilities assigned to the first three outcomes is similar to that of the original distribution—but it's assigning way too much probability-mass to the outcome "4":
+
+```
+>>> audience(reporter_2([x() for _ in range(100000)]))
+{1: 0.3971289947471831, 2: 0.20309555314968522, 3: 0.14860259032038173, 4: 0.2516540358474678}
+```
+
+So far, all of the Reporters we've imagined are still only putting one element in the inner sets of the list-of-sets that they return. But we could imagine `reporter_3`—
+
+```python
+def reporter_3(xs):
+    output = []
+    for x in xs:
+        if x == 1 or x == 4:
+            output.append({1, 4})
+        else:
+            output.append({x})
+    return output
+```
+
+Unlike `reporter_2` (which typically returned a list with _fewer_ elements than it received as input), the list returned by `reporter_3` has exactly as many elements as the list it took in. Yet this Reporter still prompts Audience to return a distribution with too many "4"s—and _unlike_ `reporter_2`, it doesn't even get the ratio of the other outcomes right, yielding disproportionately fewer "1"s compared to "2"s and "3"s than the original distribution—
+
+```
+>>> audience(reporter_3([x() for _ in range(100000)]))
+{1: 0.2808949431909106, 2: 0.24795967354776766, 3: 0.19037045927348376, 4: 0.2808949431909106}
+```
+
+Again, I've presented Audience and various possible Reporters as simple Python programs for illustration and simplicity, but the same _input-output relationships_ could be embodied as part of a more complicated system—perhaps an entire conscious mind which could talk.
+
+So now imagine our Audience as a _person_ with her own hopes and fears and ambitions ... ambitions whose ultimate fulfillment will require dedication, bravery—and meticulously careful planning based on an accurate estimate of $P(X)$, with almost no room for error.
+
+So, too, imagine each of our possible Reporters as a person: loyal, responsible—and, entirely coincidentally, the supplier of a good that Audience's careful plans call for in proportion to the value of $P(X = 4)$.
+
+When the expected frequency of "4"s fails to appear, Audience's lifework is in ruins. All of her training, all of her carefully calibrated plans, all the interminable hours of hard labor, were for nothing. She confronts Reporter in a furor of rage and grief.
+
+"You _lied_," she says through tears of betrayal, "I _trusted you_ and _you lied to me!_"
+
+The Reporter whose behavior corresponds to `reporter_2` replies, "How _dare_ you accuse me of lying?! Sure, I'm not a perfect program free from all bias, but everything I said was true—every outcome I reported corresponded to one of the $X_i$. [You can't call that misleading!](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting)"
+
+He is perfectly sincere. Nothing in his _consciousness_ reflects _intent_ to deceive Audience, any more than an eight-line Python program could be said to have such "intent." (Does a `for` loop "intend" anything? Does a conditional "care"? Of course not!)
+
+The Reporter whose behavior corresponds to `reporter_3` replies, "_Lying?!_ I told you the truth, the whole truth, and nothing but the truth: everything I saw, I reported. When I said an outcome was a oneorfour, it actually was a oneorfour. Perhaps you have a different category system, such that what _I_ think of as a 'oneorfour', appears to you to be any of several completely different outcomes, which you think my 'oneorfour' concept is conflating. If those outcomes had wildly different probabilities, if one was much more common than fou—I mean, than the other—then you'd have no way of knowing that from my report. But using language in a way _you_ dislike, is not lying. [I can define a word any way I want!](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong)"
+
+He, too, is perfectly sincere.
+
+### Commentary
+
+Much has been written on this website about reducing mental notions of "truth", "evidence", _&c._ [to the nonmental](https://www.lesswrong.com/posts/p7ftQ6acRkgo6hqHb/dreams-of-ai-design). One need not grapple with tendentious [mysteries](https://www.lesswrong.com/posts/6i3zToomS86oj9bS6/mysterious-answers-to-mysterious-questions) of "mind" or "consciousness", when so much more can be accomplished by considering systematic cause-and-effect processes [that result in](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence) the states of one physical system becoming correlated with the states of another—a "map" that reflects a "territory."
+
+The same methodology that was essential for studying truthseeking, is equally essential for studying the propagation of falsehood. If true "beliefs" are models that [make accurate predictions](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences), then _deception_ would presumably be communication that systematically results in _less_ accurate predictions (by a listener applying the same inference algorithms that would result in more accurate predictions when applied to direct observations or "honest" reports).
+
+In a peaceful world where most falsehood was due to random mistakes, there would be little to be gained by studying processes that systematically create erroneous maps. In a world of [conflict](https://slatestarcodex.com/2018/01/24/conflict-vs-mistake/), where there are [forces trying to slash your tires](https://www.lesswrong.com/posts/XTWkjCJScy2GFAgDt/dark-side-epistemology), one would do well do study these—_algorithms of deception!_
diff --git a/content/2019/being-wrong-doesnt-mean-youre-stupid-and-bad-probably.md b/content/2019/being-wrong-doesnt-mean-youre-stupid-and-bad-probably.md
index 70d84f1..9daf095 100644
--- a/content/2019/being-wrong-doesnt-mean-youre-stupid-and-bad-probably.md
+++ b/content/2019/being-wrong-doesnt-mean-youre-stupid-and-bad-probably.md
@@ -1,7 +1,38 @@
 Title: "Being Wrong Doesn't Mean You're Stupid and Bad (Probably)"
-Date: 2019-06-30 23:37
+Date: 2019-06-29 16:58
 Status: published
-Category: meta
+Category: philosophy
+Tags: rationality, Bayes-structure of the universe
 Slug: being-wrong-doesnt-mean-youre-stupid-and-bad-probably
 
-[(new post on _Less Wrong_)](https://www.lesswrong.com/posts/6dmKBjc7XarcQMRYW/being-wrong-doesn-t-mean-you-re-stupid-and-bad-probably)
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/6dmKBjc7XarcQMRYW/being-wrong-doesn-t-mean-you-re-stupid-and-bad-probably)
+
+Sometimes, people are reluctant to admit that they were wrong about something, because they're afraid that "You are wrong about this" carries inextricable connotations of "You are stupid and bad." But this behavior is, itself, wrong, for _at least_ two reasons.
+
+First, because it's evidential decision theory. The so-called "rationalist" "community" has a lot of [cached](https://www.lesswrong.com/posts/2MD3NMLBPCqPfnfre/cached-thoughts) clichés about this! A blank map does not correspond to a blank territory. [What's true is already so](https://www.lesswrong.com/posts/HYWhKXRsMAyvRKRYz/you-can-face-reality); owning up to it doesn't make it worse. Refusing to go to the doctor (thereby _avoiding encountering evidence_ that you're sick) doesn't keep you healthy.
+
+_If_ being wrong means that you're stupid and bad, then preventing yourself from _knowing_ that you were wrong doesn't stop you from being stupid and bad _in reality_. It just prevents you from _knowing_ that you're stupid and bad—which is an important fact to know (if it's true), because if you don't _know_ that you're stupid and bad, then it probably won't occur to you to even _look_ for possible interventions to make yourself _less_ stupid and _less_ bad.
+
+Second, while "You are wrong about this" _is_ evidence for the "You are stupid and bad" hypothesis if stupid and bad people are more likely to be wrong, I claim that it's _very weak_ evidence. (Although it's possible that I'm wrong about this—and if I'm wrong, it's furthermore possible that the _reason_ I'm wrong is because I'm stupid and bad.)
+
+Exactly _how weak_ evidence is it? It's hard to guess directly, but fortunately, we can use probability theory to reduce the claim into more "atomic" conditional and prior probabilities that might be easier to estimate!
+
+Let $W$ represent the proposition "You are wrong about something", $S$ represent the proposition "You are stupid", and $B$ represent the proposition "You are bad."
+
+By Bayes's theorem, the probability that you are stupid and bad given that you're wrong about something is given by—
+
+$$P(S,B|W)=\frac{P(W|S,B)P(S,B)}{P(W|S,B)P(S,B)+P(W|S, \neg B)P(S, \neg B)+P(W| \neg S,B)P( \neg S,B)+P(W| \neg S, \neg B)P( \neg S, \neg B)}$$
+
+For the purposes of this calculation, let's assume that badness and stupidity are statistically independent. I doubt this is true in the real world, but because I'm stupid and bad (at math), I want that simplifying assumption to make the algebra easier for me. That lets us unpack the conjunctions, giving us—
+
+$$P(S,B|W)=\frac{P(W|S,B)P(S)P(B)}{P(W|S,B)P(S)P(B)+P(W|S, \neg B)P(S)P(\neg B)+P(W| \neg S,B)P( \neg S)P(B)+P(W| \neg S, \neg B)P( \neg S)P(\neg B)}$$
+
+This expression has six degrees of freedom: $P(S)$, $P(B)$, $P(W|S,B)$, $P(W|S, \neg B)$, $P(W|\neg S,B)$, $P(W|\neg S, \neg B)$. Arguing about the values of these six individual parameters is probably more productive than arguing about the value of $P(S,B|W)$ directly!
+
+Suppose half the people are stupid ($P(S) = 0.5$), one-tenth of people are bad ($P(B) = 0.1$), and that most people are wrong, but that being stupid or bad each make you somewhat more likely to be wrong, to the tune of $P(W|\neg S, \neg B) = 0.8$, $P(W|S, \neg B) = P(W|\neg S,B) = 0.85$, and $P(W|S,B) = 0.9$. So our posterior probabilty that someone is stupid and bad given that they were wrong once is
+
+$$P(S,B|W) = \frac{(0.9)(0.5)(0.1)}{(0.9)(0.5)(0.1)+(0.85)(0.5)(0.9)+(0.85)(0.5)(0.1)+(0.8)(0.5)(0.9)}$$
+
+$$\approx 0.0542$$
+
+But the base rate of being stupid and bad is (0.1)(0.5) = 0.05. Learning that someone was wrong only raised our probability that they are stupid and bad by 0.0042. That's a small number that you shouldn't worry about!
diff --git a/content/2019/but-it-doesnt-matter.md b/content/2019/but-it-doesnt-matter.md
new file mode 100644
index 0000000..ae99e96
--- /dev/null
+++ b/content/2019/but-it-doesnt-matter.md
@@ -0,0 +1,10 @@
+Title: “But It Doesn’t Matter”
+Date: 2019-05-31 19:06
+Status: published
+Category: philosophy
+Tags: rationality, honesty
+Slug: but-it-doesnt-matter
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/NG4XQEL5PTyguDMff/but-it-doesn-t-matter)
+
+If you ever find yourself saying, "Even if Hypothesis _H_ is true, it doesn't have any decision-relevant implications," _you are rationalizing!_ The fact that _H_ is interesting enough for you to be considering the question at all (it's not some arbitrary trivium like the 1923th binary digit of π, or the low temperature in São Paulo on September 17, 1978) means that it must have some relevance to the things you care about. It is _vanishingly improbable_ that your optimal decisions are going to be the _same_ in worlds where _H_ is true and worlds where _H_ is false. The fact that you're tempted to _say_ they're the same is probably because some part of you is afraid of some of the imagined consequences of _H_ being true. But _H_ is already true or already false! If you happen to live in a world where _H_ is true, and you make decisions as if you lived in a world where _H_ is false, you are thereby missing out on all the extra utility you would get if you made the _H_-optimal decisions instead! If you can figure out exactly what you're afraid of, maybe that will help you work out what the _H_-optimal decisions are. Then you'll be a [better position to successfully notice](https://www.lesswrong.com/posts/3XgYbghWruBMrPTAL/leave-a-line-of-retreat) which world you _actually_ live in.
diff --git a/content/2019/firming-up-not-lying-around-its-edge-cases-is-less-broadly-useful-than-one-might-initially-think.md b/content/2019/firming-up-not-lying-around-its-edge-cases-is-less-broadly-useful-than-one-might-initially-think.md
new file mode 100644
index 0000000..d6c58e3
--- /dev/null
+++ b/content/2019/firming-up-not-lying-around-its-edge-cases-is-less-broadly-useful-than-one-might-initially-think.md
@@ -0,0 +1,107 @@
+Title: Firming Up Not-Lying Around Its Edge-Cases Is Less Broadly Useful Than One Might Initially Think
+Date: 2019-12-26 21:09
+Status: published
+Category: philosophy
+Tags: rationality, honesty
+Slug: firming-up-not-lying-around-its-edge-cases-is-less-broadly-useful-than-one-might-initially-think
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly)
+
+**Reply to**: [Meta-Honesty: Firming Up Honesty Around Its Edge-Cases](https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases)
+
+Eliezer Yudkowsky, listing advantages of a "wizard's oath" ethical code of "Don't say things that are literally false", writes—
+
+> Repeatedly asking yourself of every sentence you say aloud to another person, "Is this statement actually and literally true?", helps you build a skill for navigating out of your internal smog of not-quite-truths.
+
+I mean, that's _one_ hypothesis about the psychological effects of adopting the wizard's code.
+
+A potential problem with this is that human natural language contains a _lot_ of ambiguity. Words can be used in many ways depending on context. Even the specification "literally" in "literally false" is less useful than it initially appears when you consider that the way people _ordinarily_ speak when they're being truthful is actually pretty dense with metaphors that we typically don't _notice_ as metaphors because they're common enough to be recognized legitimate uses that all fluent speakers will understand.
+
+For example, if I want to convey the meaning that our study group has covered a lot of material in today's session, and I say, "Look how far we've come today!" it would be _pretty weird_ if you were to object, "_Liar!_ We've been in this room the whole time and haven't physically moved at all!" because in this case, it really is obvious to all ordinary English speakers that that's not what I meant by "how far we've come."
+
+Other times, the "intended"[^intended] interpretation of a statement is not only not obvious, but speakers can even mislead by motivatedly equivocating between different definitions of words: the immortal Scott Alexander has written a lot about this phenomenon under the labels ["motte-and-bailey doctrine"](https://slatestarcodex.com/2014/11/03/all-in-all-another-brick-in-the-motte/) (as [coined by Nicholas Shackel](https://philpapers.org/archive/SHATVO-2.pdf)) and ["the noncentral fallacy"](https://www.lesswrong.com/posts/yCWPkLi8wJvewPbEp/the-noncentral-fallacy-the-worst-argument-in-the-world).
+
+[^intended]: I'm scare-quoting "intended" because this process isn't necessarily conscious, and probably usually isn't. Internal distortions of reality in [imperfectly deceptive social organisms](https://intelligence.org/files/CFAI.pdf#page=48) can be [adaptive for the function of deceiving conspecifics](https://www.lesswrong.com/posts/DSnamjnW7Ad8vEEKd/trivers-on-self-deception).
+
+For example, Zvi Mowshowitz has written about how [the claim that "everybody knows" something](https://www.lesswrong.com/posts/BNfL58ijGawgpkh9b/everybody-knows)[^fake] is often used to establish fictitious social proof, or silence those attempting to tell the thing to people who really don't know, but it feels weird (to my intuition, at least) to [call it a "lie"](https://www.lesswrong.com/posts/bSmgPNS6MTJsunTzS/maybe-lying-doesn-t-exist), because the speaker can just say, "Okay, you're right that not literally[^lit] everyone knows; I meant that _most_ people know but was using a common hyperbolic turn-of-phrase and I reasonably expected you to figure that out."
+
+[^fake]: If I had written this post, I would have titled it "Fake [Common Knowledge](https://www.lesswrong.com/posts/9QxnfMYccz9QRgZ5z/the-costly-coordination-mechanism-of-common-knowledge)" (following in the tradition of ["Fake Explanations"](https://www.lesswrong.com/posts/fysgqk4CjAwhBgNYT/fake-explanations), ["Fake Optimization Criteria"](https://www.lesswrong.com/posts/i6fKszWY6gLZSX2Ey/fake-optimization-criteria), ["Fake Causality"](https://www.lesswrong.com/posts/RgkqLqkg8vLhsYpfh/fake-causality), _&c._)
+
+[^lit]: But it's worth noting that the "Is this statement actually and literally true?" test, taken literally, should have caught this, even if my intuition still doesn't want to call it a "lie."
+
+So the question "Is this statement actually and literally true?" is itself potentially ambiguous. It could mean either—
+
+ * "Is this statement actually and literally true _as the audience will interpret it?_"; or,
+ * "Does this statement _permit an interpretation under which_ it is actually and literally true?"
+
+But while the former is complicated and hard to establish, the latter is ... not necessarily that strict of a constraint in most circumstances?
+
+Think about it. When's the last time you needed to consciously tell a bald-faced, _unambiguous_ lie?—something that could realistically be _outright proven false_ in front of your peers, rather than dismissed with a "reasonable" amount of language-lawyering. (Whether "Fine" is a lie in response to "How are you?" depends on exactly what "Fine" is understood to mean in this context. ["Being acceptable, adequate, passable, or satisfactory"](https://en.wiktionary.org/wiki/fine#Adjective)—to what standard?)
+
+Maybe I'm _unusually_ honest—or possibly unusually bad at remembering when I've lied!?—but I'm not sure I even _remember_ the last time I told an outright unambiguous lie. The kind of situation where I would need to do that just _doesn't come up that often_.
+
+Now ask yourself how often your speech has been partially optimized for any function _other_ than providing listeners with information that will help them [better anticipate their experiences](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences). The answer is, "Every time you open your mouth"[^mouth]—and if you disagree, then you're lying. (Even if you only say true things, you're more likely to pick true things that make you look good, rather than your most embarrassing secrets. That's [optimization](https://www.lesswrong.com/posts/D7EcMhL26zFNbJ3ED/optimization).)
+
+[^mouth]: Actually, that's not literally true! You often open your mouth to breathe or eat without saying anything at all! Is the referent of this footnote then a blatant lie on my part?—or can I expect you to _know what I meant_?
+
+In the study of AI alignment, it's a truism that failures of alignment [can't be fixed by deontological "patches"](https://arbital.greaterwrong.com/p/patch_resistant). If your AI is exhibiting [weird and extreme](https://arbital.greaterwrong.com/p/edge_instantiation) behavior (with respect to what you _really wanted_, if not what you actually programmed), then adding a penalty term to exclude _that specific behavior_ will just result in the AI executing the "nearest unblocked" strategy, which will probably also be undesirable: [if you prevent your happiness-maximizing AI from administering heroin to humans](https://arbital.greaterwrong.com/p/nearest_unblocked#exampleproducinghappiness), it'll start administering cocaine; if you hardcode a list of banned happiness-producing drugs, it'll start researching new drugs, or just _pay_ humans to take heroin, _&c._
+
+Humans are also intelligent agents. (Um, sort of.) If you don't genuinely have the [intent to inform](http://benjaminrosshoffman.com/honesty-and-perjury/#Intent_to_inform) your audience, but consider yourself ethically bound to be honest, but your conception of _honesty_ is simply "not lying", you'll naturally gravitate towards the nearest unblocked [cognitive algorithm](https://www.lesswrong.com/posts/HcCpvYLoSFP4iAqSz/rationality-appreciating-cognitive-algorithms) [of deception](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception).[^promise]
+
+[^promise]: A similar phenomenon may occur with other attempts at ethical bindings: for example, confidentiality promises. Suppose Open Opal tends to [wear her heart on her sleeve](https://en.wiktionary.org/wiki/wear_one%27s_heart_on_one%27s_sleeve) and more specifically, believes in lies of omission: if she's talking with someone _she_ trusts, and she has information [_relevant_](https://www.lesswrong.com/posts/GSz8SrKFfW7fJK2wN/relevance-norms-or-gricean-implicature-queers-the-decoupling) to that conversation, she finds it _incredibly psychologically painful_ to _pretend not to know_ that information. If Paranoid Paris has _much_ stronger [privacy](https://www.lesswrong.com/posts/v3Nnsm5HgvEBBDpEZ/privacy) intuitions than Opal and wants to message her about a sensitive subject, Paris might demand a promise of secrecy from Opal ("Don't share the content of this conversation")—only to spark conflict later when Opal construes the literal text of the promise more narrowly than Paris might have hoped ("'Don't share the content' means don't share the _verbatim text_, right? I'm still allowed to paraphrase things Paris said and attribute them to an anonymous correspondent when I think that's relevant to whatever conversation I'm in, even though that hypothetically [leaks entropy](https://www.gwern.net/Death-Note-Anonymity) if Paris has implausibly determined enemies, right?").
+
+So _another_ hypothesis about the psychological effects of adopting the wizard's code is that—however noble your initial conscious _intent_ was—in the face of sufficiently strong incentives to deceive, you just end up accidentally training yourself to get _really good_ at misleading people with a variety of [not-technically-lying](https://www.lesswrong.com/posts/PrXR66hQcaJXsgWsa/not-technically-lying) rhetorical tactics (motte-and-baileys, false [implicatures](https://plato.stanford.edu/entries/implicature/), [stonewalling](https://www.lesswrong.com/posts/wqmmv6NraYv4Xoeyj/conversation-halters), [selective reporting](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting), [clever rationalized arguments](https://www.lesswrong.com/posts/9f5EXt8KNNxTAihtZ/a-rational-argument), [gerrymandered category boundaries](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries), _&c._), all the while congratulating yourself on how "honest" you are for never, ever emitting any "literally" "false" individual sentences.
+
+-----
+
+Ayn Rand's novel _Atlas Shrugged_[^fiction] portrays a world of crony capitalism in which politicians and businessmen claiming to act for the "common good" (and not consciously lying) are actually using force and fraud to temporarily enrich themselves while destroying the [credit-assignment mechanisms](https://www.lesswrong.com/posts/Ajcq9xWi2fmgn8RBJ/the-credit-assignment-problem) Society needs to coordinate production.[^rand]
+
+[^fiction]: I know, [fictional evidence](https://www.lesswrong.com/posts/rHBdcHGLJ7KvLJQPk/the-logical-fallacy-of-generalization-from-fictional), but I claim that the _kind of deception_ illustrated in quoted passage to follow is _entirely_ realistic.
+
+[^rand]: Okay, that's probably not _exactly_ how Rand or [her acolytes](https://www.lesswrong.com/posts/96TBXaHwLbFyeAxrg/guardians-of-ayn-rand) would put it, but that's [how I'm interpreting it](https://tvtropes.org/pmwiki/pmwiki.php/Main/DeathOfTheAuthor).
+
+In one scene, Eddie Willers (right-hand man to our railroad executive heroine Dagny Taggart) expresses horror that the government's official scientific authority, the State Science Institute, has issued a hit piece denouncing the new alloy, Rearden Metal, with which our protagonists have been planning to use to build a critical railroad line. (In actuality, we later find out, the Institute leaders want to spare themselves the embarrassment—and therefore potential loss of legislative funding—of the innovative new alloy having been invented by private industry rather than the Institute's own metallurgy department.)
+
+> "The State Science Institute," he said quietly, when they were alone in her office, "has issued a statement warning people against the use of Rearden Metal." He added, "It was on the radio. It's in the afternoon papers."
+>
+> "What did they say?"
+>
+> "Dagny, they didn't say it! ... They haven't really said it, yet it's there—and it—isn't. That's what's monstrous about it."
+>
+> [...] He pointed to the newspaper he had left on her desk. "They haven't said that Rearden Metal is bad. They haven't said it's unsafe. What they've done is ..." His hands spread and dropped in a gesture of futility.
+>
+> She saw at a glance what they had done. She saw the sentences: "It may be possible that after a period of heavy usage, a sudden fissure may appear, though the length of this period cannot be predicted. ... The possibility of a molecular reaction, at present unknown, cannot be entirely discounted. ... Although the tensile strength of the metal is obviously demonstrable, certain questions in regard to its behavior under unusual stress are not to be ruled out. ... Although there is no evidence to support the contention that the use of the metal should be prohibited, a further study of its properties would be of value."
+>
+> "We can't fight it. It can't be answered," Eddie was saying slowly. "We can't demand a retraction. We can't show them our tests or prove anything. They've said nothing. They haven't said a thing that could be refuted and embarrass them professionally. It's the job of a coward. You'd expect it from some con-man or blackmailer. But, Dagny! It's the State Science Institute!"
+
+I think Eddie is right to feel horrified and betrayed here. At the same time, it's notable that with respect to wizard's code, _no lying has taken place_.
+
+I like to imagine the statement having been drafted by an idealistic young scientist in the [moral maze](https://www.lesswrong.com/posts/45mNHCMaZgsvfDXbw/quotes-from-moral-mazes) of Dr. Floyd Ferris's office at the State Science Institute. Our scientist knows that his boss, Dr. Ferris, expects a statement that will make Rearden Metal look bad; the negative consequences to the scientist's career for failing to produce such a statement will be severe. (Dr. Ferris didn't _say_ that, but [he didn't have to](https://www.lesswrong.com/posts/9fB4gvoooNYa4t56S/power-buys-you-distance-from-the-crime).) But the lab results on Rearden Metal came back with flying colors—by every available test, the alloy is superior to steel along every dimension.
+
+Pity the dilemma of our poor scientist! On the one hand, scientific integrity. On the other hand, [the incentives](https://www.lesswrong.com/posts/5nH5Qtax9ae8CQjZ9/no-it-s-not-the-incentives-it-s-you).
+
+He decides to follow a rule that he thinks will preserve his "inner agreement with truth which allows ready recognition": after every sentence he types into his report, he will ask himself, "Is this statement actually and literally true?" For that is his mastery.
+
+Thus, his writing process goes like this—
+
+"It may be possible after a period of heavy usage, a sudden fissure may appear." Is this statement actually and literally true? _Yes!_ It [_may be possible!_](https://www.lesswrong.com/posts/ooypcn7qFzsMcy53R/infinite-certainty)
+
+"The possibility of a molecular reaction, at present unknown, cannot be entirely discounted." Is this statement actually and literally true? _Yes!_ The _possibility_ of a molecular reaction, at present unknown, _cannot be entirely discounted_. Okay, so there's [not enough](https://www.lesswrong.com/posts/nj8JKFoLSMEmD3RGp/how-much-evidence-does-it-take) evidence to [single out](https://www.lesswrong.com/posts/MwQRucYo6BZZwjKE7/einstein-s-arrogance) that possibility as [worth paying attention to](https://www.lesswrong.com/posts/X2AD2LgtKgkRNPj2a/privileging-the-hypothesis). [But there's still a chance, right?](https://www.lesswrong.com/posts/q7Me34xvSG3Wm97As/but-there-s-still-a-chance-right)
+
+"Although the tensile strength of the metal is obviously demonstrable, certain questions in regard to its behavior under unusual stress are not to be ruled out." Is this statement actually and literally true? _Yes!_ The lab tests demonstrated the metal's unprecedented tensile strength. But certain questions in regard to its behavior under unusual stress are _not to be ruled out_—the [probability isn't _zero_](https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-are-not-probabilities).
+
+And so on. You see the problem. Perhaps a member of the general public who _knew_ about the corruption at the State Science Institute could read the report and [infer the existence of hidden evidence](https://www.lesswrong.com/posts/kJiPnaQPiy4p9Eqki/what-evidence-filtered-evidence): "Wow, even when trying their hardest to trash Rearden Metal, _this_ is the worst they could come up with? Rearden Metal must be pretty great!"
+
+_But they won't_. An institution that proclaims to be dedicated to "science" is asking for a _very high_ level of trust—and [in the absence of a trustworthy auditor](https://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3F), they might get it. Science is complicated enough and natural language is ambiguous enough, that that kind of trust that can be _betrayed_ without lying.
+
+I want to emphasize that I'm _not_ saying the report-drafting scientist in the scenario I've been discussing is a "bad person." (As it is written, [almost no one is evil; almost everything is broken.](https://blog.jaibot.com/)) Under more favorable conditions—in a world where metallurgists had the academic freedom to speak the truth as they see it [(even if their voice trembles)](https://www.lesswrong.com/posts/pZSpbxPrftSndTdSf/honesty-beyond-internal-truth) without being threatened with ostracism and starvation—the _sort of person_ who finds the wizard's oath appealing, wouldn't even be _tempted_ to engage in these kinds of not-technically-lying shenanigans. But the point of the wizard's oath is to constrain you, to have a _simple_ bright-line rule to _force_ you to be truthful, _even when other people are making that genuinely difficult_. Yudkowsky's meta-honesty proposal is a clever attempt to strengthen the foundations of this ethic by formulating a more complicated theory that can account for the edge-cases under which even unusually honest people typically agree that lying is okay, usually due to extraordinary coercion by an adversary, as with the proverbial murderer or Gestapo officer at the door.
+
+And yet it's _precisely_ in adversarial situations that the wizard's oath is most constraining (and thus, arguably, most useful). You probably don't _need_ special [ethical inhibitions](https://www.lesswrong.com/posts/cyRpNbPsW8HzsxhRK/ethical-inhibitions) to tell the truth to your friends, because [you should expect to _benefit_ from friendly agents having more accurate beliefs](http://benjaminrosshoffman.com/humility-argument-honesty/).
+
+But an enemy who wants to use information to hurt you is most constrained if the worst they can do is [selectively report](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting) harmful-to-you _true_ things, rather than just making things up—and therefore, by symmetry, if you want to use information to hurt an enemy, _you_ are most constrained if the worst you can do is selectively report harmful-to-the-enemy true things, rather that just making things up.
+
+Thus, while the study of how to minimize information transfer to an adversary under the constraint of not lying is certainly _interesting_, I argue that this "firming up" is of limited practical utility given [the ubiquity](https://unstableontology.com/2019/09/10/truth-telling-is-aggression-in-zero-sum-frames/) of _other_ kinds of deception. A theory of under what conditions conscious explicit unambiguous outright lies are acceptable doesn't help very much with combating _intellectual_ dishonesty—and I fear that intellectual dishonesty, plus sufficient intelligence, is enough to destroy the world all on its own, without the help of conscious explicit unambiguous outright lies.
+
+Unfortunately, I do not, at present, have a superior alternative ethical theory of honesty to offer. I don't _know_ how to unravel the web of deceit, rationalization, excuses, disinformation, bad faith, fake news, phoniness, gaslighting, and fraud that threatens to consume us all. But one thing I'm pretty sure _won't_ help much is _clever logic puzzles about implausibly sophisticated Nazis_.
+
+_(Thanks to Michael Vassar for feedback on an earlier draft.)_
diff --git a/content/2019/heads-i-win-tails-never-heard-of-her-or-selective-reporting-and-the-tragedy-of-the-green-rationalists.md b/content/2019/heads-i-win-tails-never-heard-of-her-or-selective-reporting-and-the-tragedy-of-the-green-rationalists.md
index 9d62b83..49df0ed 100644
--- a/content/2019/heads-i-win-tails-never-heard-of-her-or-selective-reporting-and-the-tragedy-of-the-green-rationalists.md
+++ b/content/2019/heads-i-win-tails-never-heard-of-her-or-selective-reporting-and-the-tragedy-of-the-green-rationalists.md
@@ -1,8 +1,74 @@
-Title: "Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists"
+Title: Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists
 Date: 2019-09-23 21:16
 Status: published
-Category: meta
-Tags: elsewhere
+Category: philosophy
+Tags: rationality, epistemology, politics
 Slug: heads-i-win-tails-never-heard-of-her-or-selective-reporting-and-the-tragedy-of-the-green-rationalists
 
-[(new post on _Less Wrong_)](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting)
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting)
+
+**Followup to:** [What Evidence Filtered Evidence?](https://www.lesswrong.com/posts/kJiPnaQPiy4p9Eqki/what-evidence-filtered-evidence) 
+
+In ["What Evidence Filtered Evidence?"](https://www.lesswrong.com/posts/kJiPnaQPiy4p9Eqki/what-evidence-filtered-evidence), we are asked to consider a scenario involving a coin that is _either_ biased to land Heads 2/3rds of the time, _or_ Tails 2/3rds of the time. Observing Heads is 1 bit of evidence for the coin being Heads-biased (because the Heads-biased coin lands Heads with probability 2/3, the Tails-biased coin does so with probability 1/3, the likelihood ratio of these is $\frac{2/3}{1/3} = 2$, and $\log_{2} 2 = 1$), and analogously and respectively for Tails.
+
+If such a coin is flipped ten times by someone who [doesn't make literally false statements](https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases#2__The_law_of_no_literal_falsehood_), who then reports that the 4th, 6th, and 9th flips came up Heads, then the update to our beliefs about the coin depends on what _algorithm_ the not-lying[^honest] reporter used to decide to report those flips in particular. If they always report the 4th, 6th, and 9th flips _independently_ of the flip outcomes—if there's no [evidential entanglement](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence) between the flip outcomes and the choice of which flips get reported—then reported flip-outcomes can be treated the same as flips you observed yourself: three Headses is 3 * 1 = 3 bits of evidence in favor of the hypothesis that the coin is Heads-biased. (So if we were initially 50:50 on the question of which way the coin is biased, our posterior odds after collecting 3 bits of evidence for a Heads-biased coin would be $2^3:1$ = 8:1, or a probability of 8/(1 + 8) ≈ 0.89 that the coin is Heads-biased.)
+
+[^honest]: I don't quite want to use the word _honest_ here.
+
+On the other hand, if the reporter mentions only and exactly the flips that came out Heads, then we can _infer_ that the other 7 flips came out Tails (if they didn't, the reporter would have mentioned them), giving us posterior odds of $2^3:2^7$ = 1:16, or a probability of around 0.06 that the coin is Heads-biased.
+
+So far, so standard. (You _did_ [read the Sequences](https://www.readthesequences.com/), right??) What I'd like to _emphasize_ about this scenario today, however, is that while a Bayesian reasoner who _knows_ the non-lying reporter's algorithm of what flips to report will never be misled by the selective reporting of flips, a Bayesian with _mistaken_ beliefs about the reporter's decision algorithm can be misled _quite badly_: compare the 0.89 and 0.06 probabilities we just derived given the _same_ reported outcomes, but different assumptions about the reporting algorithm.
+
+If the coin gets flipped a sufficiently large number of times, a reporter whom you _trust_ to be impartial (but isn't), can _make you believe anything she wants without ever telling a single lie_, just with [appropriate selective reporting](https://slatestarcodex.com/2015/09/16/cardiologists-and-chinese-robbers/). Imagine a _very_ biased coin that comes up Heads 99% of the time. If it gets flipped ten thousand times, 100 of those flips will be Tails (in expectation), giving a selective reporter plenty of examples to point to if she wants to convince you that the coin is extremely Tails-biased.
+
+Toy models about biased coins are instructive for constructing examples with explicitly calculable probabilities, but the same _structure_ applies to any real-world situation where you're receiving evidence from other agents, and you have uncertainty about what algorithm is being used to determine what reports get to you. Reality is like the coin's bias; evidence and arguments are like the outcome of a particular flip. _Wrong_ theories [will still have _some_ valid arguments and evidence supporting them](https://www.lesswrong.com/posts/627DZcvme7nLDrbZu/update-yourself-incrementally) (as even a very Heads-biased coin will come up Tails sometimes), but theories that are [_less_ wrong](https://tvtropes.org/pmwiki/pmwiki.php/Main/TitleDrop) will have _more_.
+
+If selective reporting is mostly due to the idiosyncratic [bad intent](http://benjaminrosshoffman.com/bad-faith-behavior-not-feeling/) of rare malicious actors, then you might hope for safety in [(the law of large)](https://en.wikipedia.org/wiki/Law_of_large_numbers) numbers: if Helga in particular is systematically more likely to report Headses than Tailses that she sees, then her flip reports will diverge from everyone else's, and you can take that into account when reading Helga's reports. On the other hand, if selective reporting is mostly due to systemic _structural_ factors that result in _correlated_ selective reporting even among well-intentioned people who are being honest as best they know how,[^how] then you might have a more serious problem.
+
+[^how]: And it turns out that knowing _how_ to be honest is _much more work_ than one might initially think. You _have_ [read the Sequences](https://www.readthesequences.com/), right?!
+
+["A Fable of Science and Politics"](https://www.lesswrong.com/posts/6hfGNLf4Hg5DXqJCF/a-fable-of-science-and-politics) depicts a fictional underground Society polarized between two partisan factions, the Blues and the Greens. "[T]here is a 'Blue' and a 'Green' position on almost every contemporary issue of political or cultural importance." If human brains consistently understood [the is/ought distinction](https://plato.stanford.edu/entries/hume-moral/#io), then political or cultural alignment with the Blue or Green agenda wouldn't distort people's beliefs about reality. Unfortunately ... humans. (I'm not even going to finish the sentence.)
+
+Reality itself isn't on anyone's side, but any particular fact, argument, [sign, or portent](https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-line) might just so happen to be more easily construed as "supporting" the Blues or the Greens. The Blues want stronger marriage laws; the Greens want no-fault divorce. An [evolutionary psychologist](https://www.lesswrong.com/posts/epZLSoNvjW53tqNj9/evolutionary-psychology) investigating [effects of kin-recognition mechanisms on child abuse by stepparents](https://en.wikipedia.org/wiki/Cinderella_effect) might aspire to scientific objectivity, but being objective and _staying_ objective is _difficult_ when you're embedded in an [intelligent social web](https://www.lesswrong.com/posts/AqbWna2S85pFTsHH4/the-intelligent-social-web) in which in your work is going to be predictably championed by Blues and reviled by Greens.
+
+Let's make another toy model to try to understand the resulting distortions on the Undergrounders' collective epistemology. Suppose Reality is a coin—no, not a coin, a three-sided die,[^triangle] with faces colored blue, green, and gray. One-third of the time it comes up blue (representing a fact that is more easily construed as supporting the Blue narrative), one-third of the time it comes up green (representing a fact that is more easily construed as supporting the Green narrative), and one-third of the time it comes up gray (representing a fact that not even the worst ideologues know how to spin as "supporting" their side). 
+
+[^triangle]: For lack of an appropriate [Platonic solid](https://en.wikipedia.org/wiki/Platonic_solid) in three-dimensional space, maybe imagine tossing a triangle in two-dimensional space??
+
+Suppose each faction has social-punishment mechanisms enforcing consensus internally. [Without loss of generality](https://en.wikipedia.org/wiki/Without_loss_of_generality), take the Greens (with the understanding that everything that follows goes just the same if you swap "Green" for "Blue" and _vice versa_).[^choice] People observe rolls of the die of Reality, and can freely choose what rolls to report—except a resident of a Green city who reports more than 1 blue roll for every 3 green rolls is assumed to be a secret Blue Bad Guy, and faces increasing social punishment as their ratio of reported green to blue rolls falls below 3:1. (Reporting gray rolls is always safe.)
+
+[^choice]: As an author, I'm facing some conflicting desiderata in my color choices here. I want to say "Blues and Greens" _in that order_ for consistency with "A Fable of Science and Politics" (and other [classics from the Sequences](https://www.lesswrong.com/posts/uaPc4NHi5jGXGQKFS/blue-or-green-on-regulation)). Then when making an arbitrary choice to talk in terms of one of the factions in order to avoid cluttering the exposition, you might have expected me to say "Without loss of generality, take the Blues," because the _first_ item in a sequence ("Blues" in "Blues and Greens") is a more of a [Schelling point](https://www.lesswrong.com/posts/yJfBzcDL9fBHJfZ6P/nash-equilibria-and-schelling-points) than the second, or last, item. But I don't _want_ to take the Blues, because that color choice [has other associations](http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/) that I'm trying to avoid right now: if I said "take the Blues", I fear many readers would assume that I'm trying to directly push a partisan point about [soft censorship](https://slatestarcodex.com/2019/04/02/social-censorship-the-first-offender-model/) and [preference-falsification](https://en.wikipedia.org/wiki/Preference_falsification) social pressures in liberal/left-leaning subcultures in the contemporary United States. To be fair, it's _true_ that soft censorship and preference-falsification social pressures in liberal/left-leaning subcultures in the contemporary United States are, historically, what inspired me, personally, to write this post. It's okay for you to notice that! But I'm _trying_ to talk about the _general mechanisms_ that generate this _class_ of distortions on a Society's collective epistemology, independently of which faction or which ideology happens to be "on top" [in a particular place and time](https://www.lesswrong.com/posts/cKrgy7hLdszkse2pq/archimedes-s-chronophone). If I'm _doing my job right_, then my analogue in a ["nearby" Everett branch](https://www.lesswrong.com/posts/WqGCaRhib42dhKWRL/if-many-worlds-had-come-first) whose local subculture was as "right-polarized" as my Berkeley environment is "left-polarized", would have written a post making the _same_ arguments.
+
+The punishment is typically _informal_: there's no _official_ censorship from Green-controlled local governments, just a visible incentive gradient made out of social-media pile-ons, denied promotions, lost friends and mating opportunities, increased risk of being involuntarily committed to psychiatric prison,[^prison] _&c._ Even people who privately agree with dissident speech might participate in punishing it, the better to evade punishment themselves.
+
+[^prison]: Okay, they market themselves as psychiatric "hospitals", but let's not be confused by [misleading labels](https://www.lesswrong.com/posts/ZXuqNhMDcs6mYtb6i/the-american-system-and-misleading-labels).
+
+This scenario presents a problem for people who live in Green cities who want to make _and share_ accurate models of reality. It's impossible to report _every_ die roll (the only _1:1 scale_ map of the territory, is the territory itself), but it seems clear that the most generally useful models—the ones you would expect arbitrary AIs to come up with—aren't going to be sensitive to which facts are "blue" or "green". The reports of aspiring epistemic rationalists who are _just trying to make sense of the world_ will end up being about one-third blue, one-third green, and one-third gray, matching the distribution of the Reality die.
+
+From the perspective of ordinary nice smart Green citizens who have not been trained in [the Way](http://yudkowsky.net/rational/virtues/), these reports look _unthinkably_ Blue. Aspiring epistemic rationalists who are actually [paying attention](https://srconstantin.wordpress.com/2019/02/25/humans-who-are-not-concentrating-are-not-general-intelligences/) can easily distinguish Blue partisans from actual truthseekers,[^bias] but the [social-punishment machinery](http://benjaminrosshoffman.com/blame-games/) can't process more than [five words at a time](https://www.lesswrong.com/posts/4ZvJab25tDebB8FGE/you-have-about-five-words). The social consequences of being an _actual_ Blue Bad Guy, or just an honest nerd who doesn't know when to keep her stupid trap shut, are the same.
+
+[^bias]: Or rather, aspiring epistemic rationalists can do a _decent_ job of assessing the _extent to which_ someone is exhibiting truth-tracking behavior, or Blue-partisan behavior. Obviously, people who are _consciously_ trying to seek truth, are not necessarily going to _succeed_ at [overcoming bias](https://tvtropes.org/pmwiki/pmwiki.php/Main/TitleDrop), and attempts to correct for the "pro-Green" distortionary forces being discussed in this parable could easily veer into "pro-Blue" _over_-correction.
+
+In this scenario,[^assumptions] public opinion within a subculture or community in a Green area is constrained by the 3:1 (green:blue) "[Overton](https://en.wikipedia.org/wiki/Overton_window) ratio." In particular, under these conditions, it's _impossible to have a rationalist community_—at least the most naïve conception of such. If your marketing literature _says_, "Speak the truth, even if your voice trembles," but all the [savvy](https://slatestarcodex.com/2017/10/23/kolmogorov-complicity-and-the-parable-of-lightning/) high-status people's actual _reporting algorithm_ is, "Speak the truth, except when that would cause the local social-punishment machinery to mark me as a Blue Bad Guy and hurt me and any people or institutions I'm associated with—in which case, tell the most convenient lie-of-omission", then [smart sincere](https://www.overcomingbias.com/2009/12/the-smart-sincere-syndrome.html) idealists who have internalized your marketing literature as a moral ideal and trust the community to implement that ideal, are going to be _misled_ by the community's stated beliefs—and _confused_ at some of the pushback they get when submitting reports with a 1:1:1 blue:green:gray ratio.
+
+[^assumptions]: Please be appropriately skeptical about the real-world relevance of my made-up modeling assumptions! If it turned out that my choice of assumptions were (subconsciously) selected for the resulting conclusions about how bad evidence-filtering is, that would be really bad for the same reason that I'm claiming that evidence-filtering is really bad!
+
+Well, misled to _some_ extent—maybe not much! In the absence of an [Oracle AI](https://wiki.lesswrong.com/wiki/Oracle_AI) (or a competing rationalist community in Blue territory) to compare notes with, then it's not clear how one could get a _better_ map than trusting what the "green rationalists" say. With a few more made-up modeling assumptions, we can _quantify_ the distortion introduced by the Overton-ratio constraint, which will hopefully help develop an _intuition_ for how large of a problem this sort of thing might be in real life.
+
+Imagine that Society needs to make a decision about an Issue (like a question about divorce law or merchant taxes). Suppose that the facts relevant to making optimal decisions about an Issue are represented by nine rolls of the Reality die, and that the quality (utility) of Society's decision is proportional to the (base-two logarithm) entropy of the distribution of what facts get heard and discussed.[^entropy]
+
+[^entropy]: The entropy of a discrete probability distribution is maximized by the uniform distribution, in which all outcomes receive equal probability-mass. I only chose these "exactly nine equally-relevant facts/rolls" and "entropic utility" assumptions to make the arithmetic easy on me; a more realistic model might admit arbitrarily many facts into discussion of the Issue, but posit a distribution of facts/rolls with [diminishing marginal](https://en.wikipedia.org/wiki/Marginal_utility#Diminishing_marginal_utility) relevance to Society's decision quality.
+
+The maximum achievable decision quality is $\log_{2} 9$ ≈ 3.17.
+
+On average, Green partisans will find 3 "green" facts[^scare] and 3 "gray" facts to report, and mercilessly [stonewall](https://www.lesswrong.com/posts/wqmmv6NraYv4Xoeyj/conversation-halters) anyone who tries to report any "blue" facts, for a decision quality of $\log_{2} 6$ ≈ 2.58.
+
+[^scare]: The scare quotes around the adjective "'green'" (_&c._) when applied to the word "fact" (as opposed to a die roll outcome _representing_ a fact in our toy model) are significant! The facts aren't actually on anyone's side! We're trying to model the _distortions_ that arise from stupid humans _thinking_ that the facts are on someone's side! This is sufficiently important—and difficult to remember—that I should probably repeat it until it becomes obnoxious!
+
+On average, the Overton-constrained rationalists will report the same 3 "green" and 3 "gray" facts, but something interesting happens with "blue" facts: each individual can only afford to report one "blue" fact without blowing their Overton budget—but it doesn't have to be the _same_ fact for each person. Reports of all 3 (on average) blue rolls get to enter the public discussion, but get mentioned (cited, retweeted, _&c_.) 1/3 as often as green or gray rolls, in accordance with the Overton ratio. So it turns out that the constrained rationalists end up with a decision quality of $\frac{6}{7} \log_{2} 7 + \frac{1}{7} \log_{2} 21$ ≈ 3.03,[^calculation] _significantly_ better than the Green partisans—but still falling short of the theoretical ideal where all the relevant facts get their due attention.
+
+[^calculation]: You have three green slots, three gray slots, and three blue slots. You put three counters each on each of the green and gray slots, and one counter each on each of the blue slots. The frequencies of counters per slot is [3, 3, 3, 3, 3, 3, 1, 1, 1]. The total number of counters you put down is 3*6 + 3 = 18 + 3 = 21. To turn the frequencies into a probability distribution, you divide everything by 21, to get [1/7, 1/7, 1/7, 1/7, 1/7, 1/7, 1/21, 1/21, 1/21]. Then the entropy is $6\cdot-\frac{1}{7}\log_{2}\frac{1}{7}+3\cdot-\frac{1}{21}\log_{2}\frac{1}{21}$, which simplifies to $\frac{6}{7}\log_{2}7+\frac{1}{7}\log_{2}21$.
+
+If it's just not _pragmatic_ to expect people to defy [their incentives](https://www.lesswrong.com/posts/5nH5Qtax9ae8CQjZ9/no-it-s-not-the-incentives-it-s-you), is this the best we can do? Accept a somewhat distorted state of discourse, forever?
+
+At least one _partial_ remedy seems apparent. Recall from our original coin-flipping example that a Bayesian who _knows_ what the filtering process looks like, can take it into account and make the correct update. If you're filtering your evidence to avoid social punishment, but it's possible to clue in your fellow rationalists to your _filtering algorithm_ without triggering the social-punishment machinery—you mustn't [assume that everyone already knows](https://www.lesswrong.com/posts/BNfL58ijGawgpkh9b/everybody-knows)!—that's potentially a big win. In other words, [blatant cherry-picking is the best kind!](https://www.lesswrong.com/posts/KzAG4yWQJosmEjHe2/blatant-lies-are-the-best-kind)
diff --git a/content/2019/maybe-lying-doesnt-exist.md b/content/2019/maybe-lying-doesnt-exist.md
index 143b185..387537e 100644
--- a/content/2019/maybe-lying-doesnt-exist.md
+++ b/content/2019/maybe-lying-doesnt-exist.md
@@ -1,8 +1,99 @@
-Title: "Maybe Lying Doesn't Exist"
-Date: 2019-10-19 11:05
+Title: Maybe Lying Doesn't Exist
+Date: 2019-10-14 00:04
 Status: published
 Category: philosophy
-Tags: elsewhere
+Tags: rationality, honesty, philosophy of language
 Slug: maybe-lying-doesnt-exist
 
-[(new post on _Less Wrong_)](https://www.lesswrong.com/posts/bSmgPNS6MTJsunTzS/maybe-lying-doesn-t-exist)
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/bSmgPNS6MTJsunTzS/maybe-lying-doesn-t-exist)
+
+In ["Against Lie Inflation"](https://slatestarcodex.com/2019/07/16/against-lie-inflation/), the immortal Scott Alexander argues that the word "lie" should be reserved for knowingly-made false statements, and not used in an expanded sense that includes unconscious motivated reasoning. Alexander argues that the expanded sense draws the category boundaries of "lying" too widely in a way that would make the word less useful. The hypothesis that predicts everything predicts nothing: in order for "Kevin lied" to _mean something_, some possible states-of-affairs need to be identified as _not_ lying, so that the statement "Kevin lied" can correspond to [redistributing conserved probability mass](http://yudkowsky.net/rational/technical/) _away from_ "not lying" states-of-affairs _onto_ "lying" states-of-affairs.
+
+All of this is entirely correct. But Jessica Taylor (whose post ["The AI Timelines Scam"](https://unstableontology.com/2019/07/11/the-ai-timelines-scam/) inspired "Against Lie Inflation") wasn't arguing that _everything_ is lying; she was just using a _more_ permissive conception of lying than the one Alexander prefers, such that Alexander didn't think that Taylor's definition could stably and consistently identify non-lies.
+
+Concerning Alexander's arguments against the expanded definition, I find I have one strong objection (that appeal-to-consequences is an invalid form of reasoning for optimal-categorization questions for essentially the same reason as it is for questions of simple fact), and one more speculative objection (that our intuitive "folk theory" of lying may actually be empirically mistaken). Let me explain.
+
+(A small clarification: for myself, I notice that I _also_ tend to frown on the expanded sense of "lying". But the _reasons_ for frowning matter! People who superficially agree on a conclusion but for _different reasons_, are [not really on the same page](https://www.lesswrong.com/posts/n4ukoQzkgbAqpzqb5/argue-politics-with-your-best-friends)!)
+
+### Appeals to Consequences Are Invalid
+
+> There is no method of reasoning more common, and yet none more blamable, than, in philosophical disputes, to endeavor the refutation of any hypothesis, by a pretense of its dangerous consequences[.]
+>
+> —[David Hume](https://www.bartleby.com/37/3/12.html)
+
+Alexander contrasts the imagined consequences of the expanded definition of "lying" becoming more widely accepted, to a world that uses the restricted definition:
+
+> [E]veryone is much angrier. In the restricted-definition world, a few people write posts suggesting that there may be biases affecting the situation. In the expanded-definition world, those same people write posts accusing the other side of being liars perpetrating a fraud. I am willing to listen to people suggesting I might be biased, but if someone calls me a liar I'm going to be pretty angry and go into defensive mode. I'll be less likely to hear them out and adjust my beliefs, and more likely to try to attack them.
+
+But this is an [appeal to consequences](https://en.wikipedia.org/wiki/Appeal_to_consequences). [Appeals to consequences](https://www.lesswrong.com/posts/P3FQNvnW8Cz42QBuA/dialogue-on-appeals-to-consequences) are invalid because they represent a map–territory confusion, an attempt to optimize our _description_ of reality at the expense of our ability to describe reality _accurately_ (which we need in order to _actually_ optimize reality).
+
+(Again, the appeal is still invalid even if the conclusion—in this case, that unconscious rationalization shouldn't count as "lying"—might be true for _other reasons_.)
+
+Some aspiring epistemic rationalists like to call this the ["Litany of Tarski"](https://wiki.lesswrong.com/wiki/Litany_of_Tarski). _If_ Elijah is lying (with respect to whatever the [optimal category boundary](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries) for "lying" turns out to be according to [our standard Bayesian philosophy of language](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong)), _then_ I desire to believe that Elijah is lying (with respect to the optimal category boundary according to ... _&c._). _If_ Elijah is _not_ lying (with respect to ... _&c._), _then_ I desire to believe that Elijah is _not_ lying.
+
+If the one comes to me and says, "Elijah is not lying; to support this claim, I offer this-and-such evidence of his sincerity," then this is right and proper, and I am eager to examine the evidence presented.
+
+If the one comes to me and says, "You should choose to define _lying_ such that Elijah is not lying, because if you said that he was lying, then he might feel angry and defensive," this is _insane_. The map is not the territory! If Elijah's behavior is, _in fact_, deceptive—if he says things that cause people who trust him to be worse at [anticipating their experiences](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences) when he reasonably [could](https://www.lesswrong.com/posts/3buXtNiSK8gcRLMSG/possibility-and-could-ness) have avoided this—I can't make his behavior not-deceptive by _changing the meanings of words_.
+
+Now, I _agree_ that it might very well empirically be the case that if I _say_ that Elijah is lying (where Elijah can hear me), he might get angry and defensive, which could have a variety of negative social consequences. But that's not an argument for changing the definition of lying; that's an argument that I have an incentive to lie about whether I think Elijah is lying! (Though [Glomarizing](https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases#1__Glomarization_can_t_practically_cover_many_cases_) about whether I think he's lying might be an even better play.)
+
+Alexander is concerned that people might strategically equivocate between different definitions of "lying" as an unjust social attack against the innocent, using the classic [motte-and-bailey](https://slatestarcodex.com/2014/11/03/all-in-all-another-brick-in-the-motte/) maneuver: first, argue that someone is "lying (expanded definition)" (the motte), then switch to treating them as if they were guilty of "lying (restricted definition)" (the bailey) and hope no one notices.
+
+So, I agree that [this is a very real problem](https://www.lesswrong.com/posts/shoMpaoZypfkXv84Y/variable-question-fallacies). But it's worth noting that the problem of equivocation between [different category boundaries associated with the same word](https://www.lesswrong.com/posts/4FcxgdvdQP45D6Skg/disguised-queries) applies _symmetrically_: if it's possible to use an expanded definition of a socially-disapproved category as the motte and a restricted definition as the bailey in an unjust attack against the innocent, then it's _also_ possible to use an expanded definition as the bailey and a restricted definition as the motte in an unjust defense of the guilty. Alexander writes:
+
+> The whole reason that rebranding lesser sins as "lying" is tempting is because everyone knows "lying" refers to something very bad. 
+
+Right—and conversely, because everyone knows that "lying" refers to something very bad, it's tempting to rebrand lies as lesser sins. Ruby Bloom [explains what this looks like in the wild](https://www.lesswrong.com/posts/QB9eXzzQWBhq9YuB8/rationalizing-and-sitting-bolt-upright-in-alarm#R9kEwAz8YbQTWGPsB):
+
+> I worked in a workplace where lying was commonplace, conscious, and system 2. Clients asking if we could do something were told "yes, we've already got that feature (we hadn't) and we already have several clients successfully using that (we hadn't)." Others were invited to be part an "existing beta program" _alongside others just like them_ (in fact, they would have been the very first). When I objected, I was told "no one wants to be the first, so you have to say that."
+>
+> [...] I think they lie to themselves that they're not lying (so that if you search their thoughts, they never think "I'm lying")[.]
+
+If your interest in the philosophy of language is primarily to _avoid being blamed for things_—perhaps because you perceive that you live [in a Hobbesian dystopia](https://www.lesswrong.com/posts/YRgMCXMbkKBZgMz4M/asymmetric-justice#puGDkhWCcaNJEMkdz) where the primary function of words is to elicit actions, where the [denotative structure](https://www.lesswrong.com/posts/i2bWqSFgyFxowTKWE/actors-and-scribes-words-and-deeds) of language was [eroded by political processes](https://www.lesswrong.com/posts/8XDZjfThxDxLvKWiM/excerpts-from-a-larger-discussion-about-simulacra) long ago, and all that's left is a [standardized list of approved attacks](https://www.lesswrong.com/posts/r2dTchodfqX4o5DYH/blame-games)—in that case, it makes perfect sense to worry about "lie inflation" but not about "lie deflation." If describing something as "lying" is primarily a weapon, then applying extra scrutiny to uses of that weapon is a wise arms-restriction treaty.
+
+But if your interest in the philosophy of language is to improve and refine the uniquely human power of [vibratory telepathy](https://www.lesswrong.com/posts/SXK87NgEPszhWkvQm/mundane-magic)—to construct shared maps that reflect the territory—if you're interested in revealing what kinds of deception are _actually happening_, and why—
+
+(in short, if you are an aspiring epistemic rationalist)
+
+—then the asymmetrical fear of false-positive identifications of "lying" but not false-negatives—along with the focus on "bad actors", "stigmatization", "attacks", _&c._—just looks _weird_. What does _that_ have to do with maximizing the probability you assign to the right answer??
+
+### The Optimal Categorization Depends on the Actual Psychology of Deception
+
+> _Deception_  
+> _My life seems like it's nothing but_  
+> _Deception_  
+> _A big charade_  
+>
+> _I never meant to lie to you_  
+> _I swear it_  
+> _I never meant to play those games_
+>
+> —["Deception"](https://www.youtube.com/watch?v=kQKs0eQHZRs) by Jem and the Holograms
+
+Even if the fear of rhetorical warfare isn't a legitimate reason to avoid calling things lies (at least privately), we're still left with the main objection that "lying" is a _different thing_ from "rationalizing" or "being biased". Everyone is biased in some way or another, but to _lie_ is ["[t]o give false information intentionally with intent to deceive."](https://en.wiktionary.org/wiki/lie#Etymology_2) Sometimes it might make sense to use the word "lie" in a [noncentral](https://www.lesswrong.com/posts/yCWPkLi8wJvewPbEp/the-noncentral-fallacy-the-worst-argument-in-the-world) sense, as when we speak of "lying to oneself" or say "Oops, I lied" in reaction to being corrected. But it's important that these senses be explicitly acknowledged as noncentral and not conflated with the central case of knowingly speaking falsehood with intent to deceive—as Alexander says, conflating the two can only be to the benefit of _actual liars_.
+
+Why would anyone disagree with this obvious ordinary view, if they _weren't_ trying to get away with the sneaky motte-and-bailey social attack that Alexander is so worried about?
+
+Perhaps because the ordinary view relies an implied theory of human psychology that we have reason to believe is false? What if _conscious_ intent to deceive is typically absent in the most common cases of people saying things that (they would be capable of realizing upon being pressed) they know not to be true? Alexander writes—
+
+> So how will people decide where to draw the line [if egregious motivated reasoning can count as "lying"]? My guess is: in a place drawn by bias and motivated reasoning, same way they decide everything else. The outgroup will be lying liars, and the ingroup will be decent people with ordinary human failings.
+
+But if the word "lying" is to actually _mean something_ rather than just being a weapon, then the ingroup and the outgroup _can't both be right_. If [symmetry considerations](https://www.lesswrong.com/posts/28bAMAxhoX3bwbAKC/are-your-enemies-innately-evil) make us doubt that one group is really that much more honest than the other, that would seem to imply that _either_ both groups are composed of decent people with ordinary human failings, _or_ that both groups are composed of lying liars. The first description certainly _sounds nicer_, but as aspiring epistemic rationalists, we're _not allowed to care_ about which descriptions sound nice; we're _only_ allowed to care about which descriptions match reality.
+
+And if all of the concepts available to us in our native language fail to match reality in different ways, then we have a tough problem that may require us to innovate.
+
+The philosopher [Roderick T. Long writes](https://mises.org/library/rothbards-left-and-right-forty-years-later)—
+
+> Suppose I were to invent a new word, "zaxlebax," and define it as "a metallic sphere, like the Washington Monument." That's the definition—"a metallic sphere, like the Washington Monument." In short, I build my ill-chosen example into the definition. Now some linguistic subgroup might start using the term "zaxlebax" as though it just meant "metallic sphere," or as though it just meant "something of the same kind as the Washington Monument." And that's fine. But my definition incorporates both, and thus conceals the false assumption that the Washington Monument is a metallic sphere; any attempt to use the term "zaxlebax," meaning what I mean by it, involves the user in this false assumption.
+
+If self-deception is as ubiquitous in human life as authors such as [Robin Hanson](http://www.overcomingbias.com/tag/hypocrisy) argue (and if you're reading this blog, this should not be a new idea to you!), then the ordinary concept of "lying" may actually be analogous to Long's "zaxlebax": the standard [_intensional_ definition](https://www.lesswrong.com/posts/HsznWM9A7NiuGsp28/extensions-and-intensions) ("speaking falsehood with conscious intent to deceive"/"a metallic sphere") fails to match the most common extensional examples that we want to use the word for ("people motivatedly saying convenient things without bothering to check whether they're true"/"the Washington Monument").
+
+Arguing for this _empirical_ thesis about human psychology is beyond the scope of this post. But _if_ we live in a sufficiently Hansonian world where the _ordinary_ meaning of "lying" fails to carve reality at the joints, then authors are faced with a tough choice: either be involved in the false assumptions of the standard believed-to-be-central intensional definition, or be deprived of the use of common [expressive vocabulary](https://www.lesswrong.com/posts/H7Rs8HqrwBDque8Ru/expressive-vocabulary). As Ben Hoffman [points out in the comments](https://slatestarcodex.com/2019/07/16/against-lie-inflation/#comment-777559) to "Against Lie Inflation", an earlier Scott Alexander didn't seem shy about calling people liars in his classic 2014 post ["In Favor of Niceness, Community, and Civilization"](https://slatestarcodex.com/2014/02/23/in-favor-of-niceness-community-and-civilization/)—
+
+> Politicians lie, but not _too much_. Take the top story on Politifact Fact Check today. Some Republican claimed his supposedly-maverick Democratic opponent actually voted with Obama's economic policies 97 percent of the time. Fact Check explains that the statistic used was actually for all votes, not just economic votes, and that members of Congress typically have to have >90% agreement with their president because of the way partisan politics work. **So it's a lie, and is properly listed as one.** [bolding mine —ZMD] But it's a lie based on slightly misinterpreting a real statistic. He didn't just totally make up a number. He didn't even just make up something else, like "My opponent personally helped design most of Obama's legislation".
+
+_Was_ the politician consciously lying? Or did he (or his staffer) arrive at the [misinterpretation via unconscious motivated reasoning](https://everythingstudies.com/2019/08/19/the-prince-and-the-figurehead/) and then just not bother to scrupulously check whether the interpretation was true? And how could Alexander know?
+
+Given my current beliefs about the psychology of deception, I find myself inclined to reach for words like "motivated", "misleading", "distorted", _&c._, and am more likely to frown at uses of "lie", "fraud", "scam", _&c._ where intent is hard to establish. But even while frowning internally, I want to avoid [tone-policing](https://www.lesswrong.com/posts/bwkZD6uskCQBJDCeC/self-consciousness-wants-to-make-everything-about-itself#Tone_arguments) people whose word-choice procedures are calibrated differently from mine when I think I understand the structure-in-the-world they're trying to point to. Insisting on replacing the six instances of the phrase "malicious lies" in "Niceness, Community, and Civilization" with "maliciously-motivated false belief" would just be _worse writing_.
+
+And I _definitely_ don't want to excuse motivated reasoning as a mere ordinary human failing for which someone can't be blamed! One of the key features that distinguishes motivated reasoning from simple mistakes is the way that the former _responds to incentives_ (such as being blamed). If the [elephant in your brain](http://elephantinthebrain.com/) thinks it can _get away with lying_ just by keeping conscious-you in the dark, it should think again!
diff --git a/content/2019/relevance-norms-or-gricean-implicature-queers-the-decoupling-contextualizing-binary.md b/content/2019/relevance-norms-or-gricean-implicature-queers-the-decoupling-contextualizing-binary.md
index 51550cf..2867a24 100644
--- a/content/2019/relevance-norms-or-gricean-implicature-queers-the-decoupling-contextualizing-binary.md
+++ b/content/2019/relevance-norms-or-gricean-implicature-queers-the-decoupling-contextualizing-binary.md
@@ -1,8 +1,38 @@
-Title: "Relevance Norms; Or, Gricean Implicature Queers the Decoupling/Contextualizing Binary"
-Date: 2019-11-24 17:56
+Title: Relevance Norms; Or, Gricean Implicature Queers the Decoupling/Contextualizing Binary
+Date: 2019-11-21 22:18
 Status: published
-Category: meta
-Tags: elsewhere
+Category: philosophy
+Tags: rationality, philosophy of language
 Slug: relevance-norms-or-gricean-implicature-queers-the-decoupling-contextualizing-binary
 
-[(new post on _Less Wrong_)](https://www.lesswrong.com/posts/GSz8SrKFfW7fJK2wN/relevance-norms-or-gricean-implicature-queers-the-decoupling)
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/GSz8SrKFfW7fJK2wN/relevance-norms-or-gricean-implicature-queers-the-decoupling)
+
+**Reply to:** [Decoupling vs Contextualising Norms](https://www.lesswrong.com/posts/7cAsBPGh98pGyrhz9/decoupling-vs-contextualising-norms)
+
+Chris Leong, [following John Nerst](https://everythingstudies.com/2018/04/26/a-deep-dive-into-the-harris-klein-controversy/), distinguishes between two alleged discursive norm-sets. Under "decoupling norms", it is understood that claims should be considered in isolation; under "contextualizing norms", it is understood that those making claims should also address potential implications of those claims in context.
+
+I argue that, at best, this is a false dichotomy that fails to clarify the underlying issues—and at worst (through no fault of Leong or Nerst), the concept of "contextualizing norms" has the potential to legitimize derailing discussions for arbitrary political reasons by eliding the key question of _which_ contextual concerns are _genuinely relevant_, thereby conflating legitimate and illegitimate bids for contextualization.
+
+Real discussions adhere to what we might call "relevance norms": it is almost universally "eminently reasonable to expect certain contextual factors or implications to be addressed." Disputes arise over _which_ certain contextual factors those are, not _whether_ context matters at all.
+
+The standard academic account explaining how what a speaker means differs from what the _sentence_ the speaker said means, is H. P. Grice's theory of conversational [implicature](https://plato.stanford.edu/entries/implicature/). Participants in a conversation are expected to add neither more nor less information than is needed to make a _relevant_ contribution to the discussion.
+
+Examples abound. If I say, "I ate some of the cookies", I'm _implicating_ that I didn't eat _all_ of the cookies, because if I had, you would have expected me to say "all", not "some" (even though the decontextualized sentence "I ate some of the cookies" is, in fact, true).
+
+Or suppose you're a guest at my house, and you ask where the washing machine is, and I say it's by the stairs. If the machine then turns out to be broken, and you ask, "Hey, did you know your washing machine is broken?" and I say, "Yes", you're probably going to be pretty baffled why I didn't say "It's by the stairs, _but you can't use it because it's broken_" earlier (even though the decontextualized answer "It's by the stairs" was, in fact, true).
+
+Leong writes:
+
+> Let's suppose that blue-eyed people commit murders at twice the rate of the rest of the population. With decoupling norms, it would be considered churlish to object to such direct statements of facts. With contextualising norms, this is deserving of criticism as it risks creates a stigma around blue-eyed people.
+
+With relevance norms, objecting might or might not make sense depending on the context in which the direct statement of fact is brought up.
+
+Suppose Della says to her Aunt Judith, "I'm so excited for my third date with my new boyfriend. He has the most beautiful blue eyes!"
+
+Judith says, "Are you sure you want to go out with this man? Blue-eyed people commit murders at twice the rate of the general population."
+
+How should Della reply to this? Judith is just in the wrong here—but _not_ as a matter of a subjective choice between "contextualizing" and "decoupling" norms, and not because blue-eyed people are a sympathetic group who we wish to be seen as allied with and don't want to stigmatize. Rather, the probability of getting murdered on a date is quite low, _and_ Della already has a lot of individuating information about whether her boyfriend is likely to be a murderer from the previous two dates. Maybe ([Fermi spitballing](https://www.lesswrong.com/posts/PsEppdvgRisz5xAHG/fermi-estimates) here) the evidence of the boyfriend's eye color raises Della's probability of being murdered from one-in-a-million to one-in-500,000? Judith's bringing the possibility up _at all_ is a waste of fear in the same sense that [lotteries are said to be a waste of hope](https://www.lesswrong.com/posts/vYsuM8cpuRgZS5rYB/lotteries-a-waste-of-hope). Fearmongering about things that are almost certainly not going to happen is _uncooperative_, in [Grice's sense](https://en.wikipedia.org/wiki/Cooperative_principle)—just like it's uncooperative to tell people where to find a washing machine that doesn't work.
+
+On the other hand, if I'm making a documentary film interviewing murderers in prison and someone asks me why so many of my interviewees have blue eyes, "Blue-eyed people commit murders at twice the rate of the rest of the population" is a _completely relevant reply_. It's not clear how else I could possibly answer the question without making reference to that fact!
+
+So far, _relevance_ has been [a black box](https://www.lesswrong.com/posts/HnS6c5Xm9p9sbm4a8/grasping-slippery-things) in this exposition: unfortunately, I don't have an elegant reduction that explains what [_cognitive algorithm_](https://www.lesswrong.com/posts/HcCpvYLoSFP4iAqSz/rationality-appreciating-cognitive-algorithms) makes some facts seem "relevant" to a given discussion. But hopefully, it should now be intuitive that the determination of what context is relevant is the consideration that is, um, relevant. [Framing](https://www.lesswrong.com/posts/f886riNJcArmpFahm/noticing-frame-differences) the matter as "decouplers" (context doesn't matter!) _vs_. "contextualizers" (context matters!) is misleading because once "contextualizing norms" have been judged admissible, it becomes easy for people to motivatedly derail any [discussions they don't like](http://www.overcomingbias.com/2008/06/against-disclai.html) with endless [isolated demands](https://slatestarcodex.com/2014/08/14/beware-isolated-demands-for-rigor/) for contextualizing [disclaimers](http://www.overcomingbias.com/2008/06/against-disclai.html).
diff --git a/content/2019/schelling-categories-and-simple-membership-tests.md b/content/2019/schelling-categories-and-simple-membership-tests.md
index ea7e92c..8d32002 100644
--- a/content/2019/schelling-categories-and-simple-membership-tests.md
+++ b/content/2019/schelling-categories-and-simple-membership-tests.md
@@ -1,7 +1,96 @@
-Title: "Schelling Categories, and Simple Membership Tests"
-Date: 2019-08-25 20:34
+Title: Schelling Categories, and Simple Membership Tests
+Date: 2019-08-25 19:43
 Status: published
-Category: meta
+Category: philosophy
+Tags: rationality, philosophy of language, game theory
 Slug: schelling-categories-and-simple-membership-tests
 
-[(new post on _Less Wrong_)](https://www.lesswrong.com/posts/edEXi4SpkXfvaX42j/schelling-categories-and-simple-membership-tests)
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/edEXi4SpkXfvaX42j/schelling-categories-and-simple-membership-tests)
+
+**Followup to**: [Where to Draw the Boundaries?](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries)
+
+_Or there might be social or psychological forces anchoring word usages on identifiable Schelling points that are easy for different people to agree upon, even at the cost of some statistical "fit"_ ...
+
+The one comes to you and says, "That paragraph about Schelling points sounded interesting. What did you mean by that? Can you give an example?"
+
+Sure. Previously on _Less Wrong_, in ["The Univariate Fallacy"](https://www.lesswrong.com/posts/cu7YY7WdgJBs3DpmJ/the-univariate-fallacy), we studied points sampled from two multivariate probability distributions $P_A$ and $P_B$, and showed that it was possible to [infer with very high probability](http://zackmdavis.net/blog/2019/05/the-typical-set/) which distribution a given point was sampled from, despite significant overlap in the marginal distributions for any one variable considered individually.
+
+From the standpoint of ["the way to carve reality at its joints, is to draw your boundaries around concentrations of unusually high probability density in Thingspace"](https://www.lesswrong.com/posts/yLcuygFfMfrfK8KjF/mutual-information-and-density-in-thingspace), the correct categorization of the points in that example is clear. We have two clearly distinguishable clusters. The [conditional independence](https://www.lesswrong.com/posts/gDWvLicHhcMfGmwaK/conditional-independence-and-naive-bayes) property is satisfied: _given_ a point's cluster-membership, knowing one of the $x_i$ doesn't tell you anything about $x_j$ for _j_ ≠ _i_. So we should draw a category boundary around each cluster. Obviously. We might ask [hypophorically](https://en.wikipedia.org/wiki/Hypophora): what could _possibly_ change this moral?
+
+More constraints on the problem, that's what!
+
+Suppose you needed to _coordinate_ with someone else to make decisions about these points—that is, it's important not just that you and your partner make good decisions, but also [that you make the _same_ decision](https://www.lesswrong.com/posts/9QxnfMYccz9QRgZ5z/the-costly-coordination-mechanism-of-common-knowledge#Coordination_Problems)—but that each of you only got to observe one coordinate from each point. As we saw, the predictive work we get from category-membership in this scenario is [spread across _many_ variables](https://marcodgdotnet.files.wordpress.com/2018/04/delgiudice_2017_heterogeneity_d_mbr.pdf): if you only get to observe a _few_ dimensions, you have a lot of uncertainty about cluster-membership (which carries over into additional uncertainty about the other dimensions that you haven't observed, but which affect the _ex post_ quality of your decision).
+
+If you and your partner were both ideal Bayesian calculators who could communicate costlessly, you would [share your observations](https://www.overcomingbias.com/2009/02/share-likelihood-ratios-not-posterior-beliefs.html), work out the correct probability, and use that to make optimal decisions. But suppose you _couldn't_ do that—either because communication is expensive, or your partner was bad at math, or any other reason. Then it would be sad if you happened to see $x_9$ = 2 and said "It's an A (probably)!", and your partner happened to see $x_{27}$ = 3 and said "It's a B (I think)!", and the two of you made inconsistent decisions.
+
+Okay, _now_ suppose that there's actually a forty-first, binary, variable that I didn't tell you about earlier, distributed like so:
+
+$$P_A(x_{41}) = \begin{cases} 3/4 & x_{41} = 0 \\ 1/4 & x_{41} = 1 \\ \end{cases}$$
+
+$$P_B(x_{41}) = \begin{cases} 1/4 & x_{41} = 0 \\ 3/4 & x_{41} = 1 \\ \end{cases}$$
+
+Observing $x_{41}$ gives you $\log_2 3$ ≈ 1.585 bits of evidence about cluster-membership, which is more than the
+
+$$\frac{1/4 + 1/16}{2} \cdot |\log_2(4)| + \frac{7/16 + 1/4}{2} \cdot |\log_2(7/4)| + \frac{1/4 + 7/16}{2} \cdot |\log_2(4/7)| + \frac{1/16 + 1/4}{2} \cdot |\log_2(4)|$$
+
+≈ 1.18 bits you can get from any one observation of one of the $x_i$ for _i_ ∈ {1...40}.
+
+If you and your partner can both observe $x_{41}$, you might end up wanting to base your shared categories and language on _that_—calling a point an "A" if it has $x_{41}$ = 0, even though such points actually came from $P_B$ a full quarter of the time—even if $x_{41}$ itself has no effect on the quality of your decisions, and what you _actually_ care about is wholely determined by the values of $x_1$ through $x_{40}$! It's not the [intension](https://www.lesswrong.com/posts/HsznWM9A7NiuGsp28/extensions-and-intensions) you would pick if you could make (and share) _more observations_—but _ex hypothesi_, you can't.
+
+If you and your partner only get to observe one variable, $x_{41}$ is your best choice—the single variable that gives you the most information about the "natural" cluster-membership. That also makes it a [_Schelling point_](https://www.lesswrong.com/posts/yJfBzcDL9fBHJfZ6P/nash-equilibria-and-schelling-points)—if you and your partner didn't get to commmunicate in advance about how you want to draw your shared category boundaries, you could pick $x_{41}$ as your defining observation and be pretty confident your partner would make the same choice. We could imagine an even more pessimistic scenario in which the Schelling point category definition (a set of variables that "stuck out" from all the others) was _less_ predictive than some other candidates—but if you couldn't _coordinate_ to _pick_ one of the more predictive category systems, you might be stuck with the Schelling point.
+
+In conclusion, the right categories to use _given_ constraints on communication and observation, might be different from the category boundaries you would draw from a "God's eye view", in part because consideration of which categories are easy for different agents to _coordinate_ on is relevant, not just raw information-theoretic expressive power. Thus, "Schelling categories."
+
+Thanks for reading!
+
+-------
+
+The one says, "No, I meant, like, a _real world_ example, not some dumb math thing for nerds. What is this post _really_ about?"
+
+It's about ... math? Or like, the relationship between math and human natural language? Like, I was wondering what "second-order" caveats or complications there might be to the basic ["carve reality at the joints" moral](https://www.lesswrong.com/posts/d5NyJ2Lf6N22AD9PB/where-to-draw-the-boundary) of our [standard Bayesian philosophy of language](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong), and some of the people I've been collaborating with lately had been talking a lot about the importanace of [_intersubjective_ epistemology](http://benjaminrosshoffman.com/humility-argument-honesty/)—that is, [_shared_ mapmaking](https://twitter.com/jessi_cata/status/1113677758071070720), so—
+
+"But where's the _actionable takeaway_? What's your _real_ agenda here, huh?"
+
+Oh. One of _those_ readers, I see. Fine, I can probably think of some—how do you say?—"applications."
+
+Ummmm ...
+
+Let's see ...
+
+Okay, here's something, maybe. What's the deal with the [age of majority](https://en.wikipedia.org/wiki/Age_of_majority)?
+
+Society needs to decide who it wants to be allowed to vote, stand trial, sign contracts, serve in the military, _&c._ Whether it's a good idea for a particular person to have these privileges presumably depends on various _relevant_ features of that person: things like cognitive ability, foresight, wisdom, relevant life experiences, _&c_. In particular, it would be _pretty weird_ for someone's fitness to vote to directly depend on _how many times the Earth has gone around the sun since they were born_. What does _that_ number have to do with anything?
+
+It doesn't! But if Society isn't well-coordinated enough to _agree_ on the exact prerequisites for voting and how to measure them, but _can_ agree that most twenty-five-year-olds have them and most eleven-year-olds don't, then we end up choosing some arbitrary age cutoff as the criterion for our "legal adulthood" social construct. It _works_, but it's just a legal fiction—and not necessarily a particularly good fiction, as any bright teenagers reading this will doubtlessly attest.
+
+If I told you that a particular fourteen-year-old was very "mature", that's a contentful statement: we have shared meaning attached to the word _mature_, such that my describing someone that way [constrains your anticipations](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences). But it's a _really complicated_ meaning, a statistical signal in behavior that your brain can pick up on, but which isn't particularly verifiable to others who might have reasons to doubt my character assessment. In contrast, age is easy for everyone to agree on. We could _imagine_ some hypothetical science-fictional Society that used brain scans and some sophisticated machine-learning classifer to determine which citizens get which privileges—but in our dumber, poorer world, calendars and subtraction will have to do.
+
+In terms of [Scott Garrabrant's taxonomy](https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy) of applications of [Goodhart's law](https://www.lesswrong.com/posts/YtvZxRpZjcFNwJecS/the-importance-of-goodhart-s-law), this is regressional Goodhart: Society _wants_ to select for maturity, chooses age as a proxy, and in the process, ends up granting or withholding privileges that a more discriminating Society maybe wouldn't.
+
+The age of majority is a case of replacing a _complicated_, [illegible](https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-called-legibility/) category ("maturity", the kind of abstract thing you might want to model as a cluster in a forty- or forty-one-dimensional space) with a _simple_ membership test (an age cutoff that everyone knows how to compute). Different people might make make different _subjective_ (but not [arbitrary](https://www.lesswrong.com/posts/HacgrDxJx3Xr7uwCR/arbitrary)) judgements of the complicated, illegible category, so in order to get a more intersubjectively robust verdict on category-membership, we rely on an _objective_ measurement that everyone can agree on.
+
+If no convenient objective measurement is available, another strategy is possible: we can _delegate_ to some canonical trusted authority, whose opinion of the complicated category will take precdence over everyone else's. An example of this is [commodity grading standards](https://www.usda.gov/our-agency/about-usda/laws-and-regulations/commodity-standards-and-grades). What is a "Grade AA" egg? Well, there's a complicated definition written down in [a manual somewhere](https://www.ams.usda.gov/sites/default/files/media/Egg%20Grading%20Manual.pdf) that you could try applying yourself—but for most people, Grade AA eggs are simply "those which have been certified as Grade AA by the USDA."[^america]
+
+[^america]: Or the analogous agency in your country.
+
+It's even possible for the "simple objective measurement" and "delegate to an authority's subjective judgement" strategies to be combined. In ["The Ideology Is Not the Movement"](https://slatestarcodex.com/2016/04/04/the-ideology-is-not-the-movement/), the immortal Scott Alexander writes about his model of the genesis of social groups—
+
+> Pre-existing differences are the raw materials out of which tribes are made. A good tribe combines people who have similar interests and styles of interaction _even before_ the ethnogenesis event. Any description of these differences will necessarily involve stereotypes, but a lot of them should be hard to argue. [...] There are subtle habits of thought, not yet described by any word or sentence, which atheists are more likely to have than other people. [...]
+>
+> The rallying flag is the explicit purpose of the tribe. It's usually a belief, event, or activity that get people with that specific pre-existing difference together and excited. Often it brings previously latent differences into sharp relief. People meet around the rallying flag, encounter each other, and say "You seem like a kindred soul!" or "I thought I was the only one!" Usually it suggests some course of action, which provides the tribe with a purpose.
+
+Eliezer Yudkowsky's ["A Fable of Science and Politics"](https://www.lesswrong.com/posts/6hfGNLf4Hg5DXqJCF/a-fable-of-science-and-politics) depicts a fictional underground society split between two such tribes: an predominantly urban tribe that believes that the unseen sky is blue (and favors an income tax, strong marriage laws, and an Earth-centric cosmology), and predominanty rural one that believes that the sky is green (and favors merchant taxes, no-fault divorce, and a heliocentric cosmology). In this story, beliefs about the color of the sky are functioning as the "rallying flag" for tribe-formation in Alexander's model—and as a Schelling point for category definition.
+
+We don't know how to _talk_ about the preëxisting undefinable habits of thought that make social groups work—it's hard to _explicitly_ articulate what exact statistical regularity our brains have detected in five-and-more-dimensional locale/sky-belief/tax-belief/divorce-belief/cosmology/_&c_.-space. (Although we could imagine some hypothetical science-fictional Society that _did_ know how to articulate it, and consequently had richer forms of social and political organization than our own.) It's a lot simpler to talk about whether someone has pledged allegiance to the rallying flag: just _ask_ someone, "What color do you believe the sky is?" (using sky-beliefs as as an "objective" simple membership test), or simply, "Are you a Blue or a Green?" (delegating the classification problem to _the person themselves_ as the authority whose discernment is to be trusted)—and whatever they say, that's what they are.
+
+Well, _probably_. We've seen that objective measurements like age are subject to regressional Goodhart, but the delegation-to-authority strategy is furthermore subject to [_adversarial_ Goodhart](https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy): once a category-membership test has been established, some agents might have an incentive to create examples that _pass the test_, but don't have the complicated, illegible properties than made the test a useful proxy in the first place.
+
+We've seen this, for example, with [title inflation](http://benjaminrosshoffman.com/excerpts-from-a-larger-discussion-about-simulacra/): we expect the "job title" (the words that get printed on business cards or immigration sponsorship forms) to be the canonical description of what someone "does", even if the vagaries of the workday encompass many tasks,[^booth] and an alien anthropologist tasked with observing the worksite and summarizing what each of the humans did might slice up her observations into categories with little resemblance to the company's formal org chart. But since we don't know _how_ to do [the obvious thing](http://zackmdavis.net/blog/2014/06/standard-advice/) and average over all possible alien anthropologists [weighted by simplicity](https://arbital.greaterwrong.com/p/1hh?l=1hh), we can only rely on the org chart—which people have [political incentives](https://www.lesswrong.com/posts/45mNHCMaZgsvfDXbw/quotes-from-moral-mazes) to manipulate, with the result that [everyone in the finance industry is a "vice president"](https://www.thebalancecareers.com/job-titles-1287163) of some sort or another.
+
+[^booth]: When I worked in [a supermarket](https://www.safeway.com/), two days a week I did Tracy's bookkeeping/customer-service job while Tracy had her weekend, which entailed counting the money from last night's tills _and_ swapping in new coinmags _and_ completing the FSM report _and_ answering the phone _and_ selling money orders _and_ covering the floral stand when the floral lady was on lunch, _&c._ I'm actually not sure what official name this role had in Safeway's official org chart. We just called it "the booth."
+
+But "Vice President" has a _literal meaning_. Or it _used to_. _Vice_, ["in place of; subordinate to."](https://en.wiktionary.org/wiki/vice#Etymology_3) _President_, one who _presides_ over some deliberative body. The adversarial-Goodhart pressures on language ["exploit[ ] the trust we have in a functioning piece of language until it's lost all meaning"](https://slatestarcodex.com/2019/07/16/against-lie-inflation/).
+
+So for readers who demand a takeaway beyond just an edge case in the math, perhaps take away this: coordination is [_costly_](https://www.lesswrong.com/posts/9QxnfMYccz9QRgZ5z/the-costly-coordination-mechanism-of-common-knowledge). From the standpoint of language as an AI capability, the social constructions that feeble humans need in order to work together may be [unavoidably dumbed-down for mass consumption](https://www.lesswrong.com/posts/4ZvJab25tDebB8FGE/you-have-about-five-words), but that's no reason to not _aspire_ to the true precision of the [Bayes-structure](https://www.lesswrong.com/posts/QrhAeKBkm2WsdRYao/searching-for-bayes-structure) to whatever extent possible.
+
+_(Thanks to Ben Hoffman for the etymology of "Vice President.")_
diff --git a/content/2019/stupidity-and-dishonesty-explain-each-other-away.md b/content/2019/stupidity-and-dishonesty-explain-each-other-away.md
new file mode 100644
index 0000000..1eacf61
--- /dev/null
+++ b/content/2019/stupidity-and-dishonesty-explain-each-other-away.md
@@ -0,0 +1,28 @@
+Title: Stupidity and Dishonesty Explain Each Other Away
+Date: 2019-12-28 11:21
+Status: published
+Category: philosophy
+Tags: rationality, honesty
+Slug: stupidity-and-dishonesty-explain-each-other-away
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/y4bkJTtG3s5d6v36k/stupidity-and-dishonesty-explain-each-other-away)
+
+The _explaining-away effect_ (or, collider bias; or, Berkson's paradox) is a statistical phenomenon in which statistically independent causes with a common effect become anticorrelated when conditioning on the effect.
+
+In the language of [_d_-separation](https://en.wikipedia.org/wiki/Bayesian_network#d-separation), if you have a [causal graph](https://www.lesswrong.com/posts/hzuSDMx7pd2uxFc5w/causal-diagrams-and-causal-models) X → Z ← Y, then conditioning on Z unblocks the path between X and Y.
+
+Daphne Koller and Nir Friedman give an example of reasoning about disease etiology: if you have a sore throat and cough, and aren't sure whether you have the flu or [mono](https://en.wikipedia.org/wiki/Infectious_mononucleosis), you should be relieved to find out it's "just" a flu, because that decreases the probability that you have mono. You _could_ be inflected with both the influenza and mononucleosis viruses, but if the flu is completely sufficient to explain your symptoms, there's no _additional_ reason to expect mono.[^koller-and-friedman]
+
+[^koller-and-friedman]: Daphne Koller and Nier Friedman, _Probabilistic Graphical Models: Principles and Techniques_, §3.2.1.2 "Reasoning Patterns"
+
+Judea Pearl gives an example of reasoning about a burglar alarm: if your neighbor calls you at your dayjob to tell you that your burglar alarm went off, it could be because of a burglary, or it could have been a false-positive due to a small earthquake. There _could_ have been both an earthquake _and_ a burglary, but if you get news of an earthquake, you'll stop worrying so much that your stuff got stolen, because the earthquake alone was sufficient to explain the alarm.[^pearl]
+
+[^pearl]: Judea Pearl, _Probabilistic Reasoning in Intelligent Systems_, §2.2.4 "Multiple Causes and 'Explaining Away'"
+
+Here's another example: if someone you're arguing with is wrong, it could be either because they're just too stupid to get the right answer, or it could be because they're being dishonest—or some combintation of the two, but more of one means that less of the other is required to explain the observation of the person being wrong. As a causal graph—[^code]
+
+[^code]: Thanks to Daniel Kumor for [example $\LaTeX$ code for causal graphs](https://dkumor.com/posts/technical/2018/08/15/causal-tikz/).
+
+![stupidity → wrongness ← dishonesty](https://i.imgur.com/aAblF44.png)
+
+Notably, the decomposition still works whether you count subconscious motivated reasoning as "stupidity" or "dishonesty". (Needless to say, it's also symmetrical across persons—if _you're_ wrong, it could be because _you're_ stupid or are being dishonest.)
diff --git a/content/2019/the-univariate-fallacy.md b/content/2019/the-univariate-fallacy.md
index 2e0036f..b78c94c 100644
--- a/content/2019/the-univariate-fallacy.md
+++ b/content/2019/the-univariate-fallacy.md
@@ -1,7 +1,103 @@
-Title: "The Univariate Fallacy"
-Date: 2019-06-15 14:53
+Title: The Univariate Fallacy
+Date: 2019-06-15 14:43
 Status: published
-Category: meta
+Category: mathematics
+Tags: statistics, rationality
 Slug: the-univariate-fallacy
 
-[(new post on _Less Wrong_)](https://www.lesswrong.com/posts/cu7YY7WdgJBs3DpmJ/the-univariate-fallacy-1)
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/cu7YY7WdgJBs3DpmJ/the-univariate-fallacy-1)
+
+There's this statistical phenomenon where it's possible for two multivariate distributions to overlap along any one variable, but be cleanly separable when you look at the entire [configuration space](https://www.lesswrong.com/posts/WBw8dDkAWohFjWQSk/the-cluster-structure-of-thingspace) at once. This is perhaps easiest to see with an illustrative diagram—
+
+![](https://i.imgur.com/Q5Gzfp3.png)
+
+The denial of this possibility (in arguments of the form, "the distributions overlap along this variable, therefore you can't say that they're different") is sometimes called the "univariate fallacy." (Eliezer Yudkowsky [proposes "covariance denial fallacy" or "cluster erasure fallacy"](https://twitter.com/ESYudkowsky/status/1124757043997372416) as potential alternative names.)
+
+Let's make this more concrete by making up an example with actual numbers instead of just a pretty diagram. Imagine we have some datapoints that live in the forty-dimensional space {1, 2, 3, 4}⁴⁰ that are sampled from one of two probability distibutions, which we'll call $P_A$ and $P_B$.
+
+For simplicity, let's suppose that the individual variables _x₁_, _x₂_, ... _x₄₀_—the coördinates of a point in our forty-dimensional space—are statistically independent. For every individual $x_i$, the marginal distribution of $P_A$ is—
+
+$$P_A(x_i) = \begin{cases} 1/4 & x_i = 1 \\ 7/16 & x_i = 2 \\ 1/4 & x_i = 3 \\ 1/16 & x_i = 4 \\ \end{cases}$$
+
+And for $P_B$—
+
+$$P_B(x_i) = \begin{cases} 1/16 & x_i = 1 \\ 1/4 & x_i = 2 \\ 7/16 & x_i = 3 \\ 1/4 & x_i = 4 \\ \end{cases}$$
+
+If you look at any one $x_i$-coördinate for a point, you can't be confident which distribution the point was sampled from. For example, seeing that _x₁_ takes the value 2 gives you a 7/4 (= 1.75) likelihood ratio in favor of that the point having been sampled from $P_A$ rather than $P_B$, which is log₂(7/4) ≈ 0.807 [bits of evidence](http://yudkowsky.net/rational/technical/).
+
+That's ... not a whole lot of evidence. If you guessed that the datapoint came from $P_A$ based on that much evidence, you'd be wrong about 4 times out of 10. (Given equal (1:1) prior odds, an odds ratio of 7:4 amounts to a probability of (7/4)/(1 + 7/4) ≈ 0.636.)
+
+And yet if we look at _many_ variables, we can achieve _supreme, godlike_ confidence about which distribution a point was sampled from. _Proving_ this is left as an exercise to the particularly intrepid reader, but a concrete _demonstration_ is probably simpler and should be pretty convincing! Let's write some Python code to sample a point **x⃗** ∈ {1, 2, 3, 4}⁴⁰ from $P_A$—
+
+```
+import random
+
+def a():
+    return random.sample(
+        [1]*4 +  # 1/4
+        [2]*7 +  # 7/16
+        [3]*4 +  # 1/4
+        [4],     # 1/16
+        1
+    )[0]
+
+x = [a() for _ in range(40)]
+print(x)
+```
+
+Go ahead and run the code yourself. (With an [online REPL](https://repl.it/languages/python3) if you don't have Python installed locally.) You'll _probably_ get a value of `x` that "looks something like"
+
+```
+[2, 1, 2, 2, 1, 1, 2, 2, 1, 2, 1, 4, 4, 2, 2, 3, 3, 1, 2, 2, 2, 4, 2, 2, 1, 2, 1, 4, 3, 3, 2, 1, 1, 3, 3, 2, 2, 3, 3, 4]
+```
+
+If someone off the street just handed you this **x⃗** without telling you whether she got it from $P_A$ or $P_B$, how would you compute the probability that it came from $P_A$?
+
+Well, because the coördinates/variables are statistically independent, you can just tally up (multiply) the individual likelihood ratios from each variable. That's only a little bit more code—
+
+```
+import logging
+
+logging.basicConfig(level=logging.INFO)
+
+def odds_to_probability(o):
+    return o/(1+o)
+
+def tally_likelihoods(x, p_a, p_b):
+    total_odds = 1
+    for i, x_i in enumerate(x, start=1):
+        lr = p_a[x_i-1]/p_b[x_i-1]  # (-1s because of zero-based array indexing)
+        logging.info("x_%s = %s, likelihood ratio is %s", i, x_i, lr)
+        total_odds *= lr
+    return total_odds
+
+print(
+    odds_to_probability(
+        tally_likelihoods(
+            x,
+            [1/4, 7/16, 1/4, 1/16],
+            [1/16, 1/4, 7/16, 1/4]
+        )
+    )
+)
+```
+
+If you run that code, you'll _probably_ see "something like" this—
+
+```
+INFO:root:x_1 = 2, likelihood ratio is 1.75
+INFO:root:x_2 = 1, likelihood ratio is 4.0
+INFO:root:x_3 = 2, likelihood ratio is 1.75
+INFO:root:x_4 = 2, likelihood ratio is 1.75
+INFO:root:x_5 = 1, likelihood ratio is 4.0
+[blah blah, redacting some lines to save vertical space in the blog post, blah blah]
+INFO:root:x_37 = 2, likelihood ratio is 1.75
+INFO:root:x_38 = 3, likelihood ratio is 0.5714285714285714
+INFO:root:x_39 = 3, likelihood ratio is 0.5714285714285714
+INFO:root:x_40 = 4, likelihood ratio is 0.25
+0.9999936561215961
+```
+
+Our computed probability that **x⃗** came from $P_A$ has several nines in it. Wow! That's pretty confident!
+
+Thanks for reading!
diff --git a/content/2019/where-to-draw-the-boundaries.md b/content/2019/where-to-draw-the-boundaries.md
index a5a77ea..c558f21 100644
--- a/content/2019/where-to-draw-the-boundaries.md
+++ b/content/2019/where-to-draw-the-boundaries.md
@@ -1,7 +1,156 @@
-Title: "Where to Draw the Boundaries?"
-Date: 2019-04-13 16:27
+Title: Where to Draw the Boundaries?
+Date: 2019-04-13 14:34
 Status: published
-Category: meta
+Category: philosophy
+Tags: rationality, philosophy of language
 Slug: where-to-draw-the-boundaries
 
-[(new post on _Less Wrong_)](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries)
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries)
+
+**Followup to:** [Where to Draw the Boundary?](https://www.lesswrong.com/posts/d5NyJ2Lf6N22AD9PB/where-to-draw-the-boundary)
+
+_Figuring where to cut reality in order to carve along the joints—figuring which things are similar to each other, which things are clustered together:_ this _is the problem worthy of a rationalist. It is what people_ should _be trying to do, when they set out in search of the floating essence of a word._
+
+_Once upon a time it was thought that the word "fish" included dolphins ..._
+
+The one comes to you and says:
+
+> The list: `{salmon, guppies, sharks, dolphins, trout}` is just a list—you can't say that a list is _wrong_. You draw category boundaries in specific ways to capture tradeoffs you care about: sailors in the ancient world wanted a word to describe the swimming finned creatures that they saw in the sea, which included salmon, guppies, sharks—and dolphins. That grouping may not be the one favored by modern evolutionary biologists, but an alternative categorization system is not an error, and borders are not objectively true or false. You're not standing in defense of truth if you insist on a word, brought explicitly into question, being used with some particular meaning. So my definition of _fish_ cannot possibly be 'wrong,' as you claim. I can define a word any way I want—in accordance with my values!
+
+So, there is a legitimate complaint here. It's true that sailors in the ancient world had a legitimate reason to want a word in their language whose [extension](https://www.lesswrong.com/posts/HsznWM9A7NiuGsp28/extensions-and-intensions) was `{salmon, guppies, sharks, dolphins, ...}`. (And modern scholars writing a translation for present-day English speakers might even translate that word as _fish_, because [most](https://www.lesswrong.com/posts/4mEsPHqcbRWxnaE5b/typicality-and-asymmetrical-similarity) members of that category are what we would call fish.) It indeed would not necessarily be helping the sailors to tell them that they need to exclude dolphins from the extension of that word, and instead include dolphins in the extension of their word for `{monkeys, squirrels, horses ...}`. Likewise, most modern biologists have little use for a word that groups dolphins and guppies together.
+
+When rationalists say that definitions can be wrong, we don't mean that there's a _unique_ category boundary that is the True floating essence of a word, and that all other possible boundaries are wrong. We mean that in order for a proposed category boundary to _not_ be wrong, it needs to capture some statistical structure in reality, even if [reality is surprisingly detailed](http://johnsalvatier.org/blog/2017/reality-has-a-surprising-amount-of-detail) and there can be _more than one_ such structure.
+
+The reason that the sailor's concept of _water-dwelling animals_ isn't necessarily wrong (at least within a particular domain of application) is because dolphins and fish actually _do_ have things in common [due to convergent evolution](http://www.brooklyn.cuny.edu/bc/ahp/LAD/C21/C21_Convergent.html), despite their differing ancestries. If we've been told that "dolphins" are water-dwellers, we can _correctly_ [predict](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences) that they're likely to have fins and a hydrodynamic shape, even if we've never seen a dolphin ourselves. On the other hand, if we predict that dolphins probably lay eggs because 97% of known fish species are oviparous, we'd get the _wrong answer_.
+
+A standard technique for understanding why some objects belong in the same "category" is to [(pretend that we can)](http://zackmdavis.net/blog/2013/05/dimensionality/) visualize objects as [existing in a very-high-dimensional configuration space](https://www.lesswrong.com/posts/WBw8dDkAWohFjWQSk/the-cluster-structure-of-thingspace), but this "Thingspace" isn't particularly well-defined: we want to map every property of an object to a dimension in our abstract space, but it's not clear how one would enumerate all possible "properties." But this isn't a major concern: we can form a space with _whatever_ properties or variables we happen to be interested in. Different choices of properties correspond to different [cross sections](https://en.wikipedia.org/wiki/Cross_section_(geometry)) of the grander Thingspace. Excluding properties from a collection would result in a "thinner", lower-dimensional [subspace](https://en.wikipedia.org/wiki/Linear_subspace) of the space defined by the original collection of properties, which would in turn be a subspace of grander Thingspace, just as a line is a subspace of a plane, and a plane is a subspace of three-dimensional space.
+
+Concerning dolphins: there would be a cluster of water-dwelling animals in the subspace of dimensions that water-dwelling animals are similar on, and a cluster of mammals in the subspace of dimensions that mammals are similar on, and dolphins would belong to _both_ of them, just as the vector [1.1, 2.1, 9.1, 10.2] in the four-dimensional vector space ℝ⁴ is simultaneously close to [1, 2, 2, 1] in the subspace [spanned](https://en.wikipedia.org/wiki/Linear_span) by _x₁_ and _x₂_, _and_ close to [8, 9, 9, 10] in the subspace spanned by _x₃_ and _x₄_.
+
+Humans are already functioning intelligences (well, sort of), so the categories that humans propose of their own accord won't be _maximally_ wrong: no one would try to propose a word for "configurations of matter that match any of these 29,122 five-megabyte descriptions but have no other particular properties in common." (Indeed, because we are [not-superexponentially-vast](https://www.lesswrong.com/posts/82eMd5KLiJ5Z6rTrr/superexponential-conceptspace-and-simple-words) minds that evolved to function in a simple, ordered universe, it actually takes some ingenuity to construct a category _that_ wrong.)
+
+This leaves aspiring instructors of rationality in something of a predicament: in order to _teach_ people how categories can be more or (ahem) [less wrong](https://tvtropes.org/pmwiki/pmwiki.php/Main/TitleDrop), you need some sort of illustrative example, but since the most natural illustrative examples won't be _maximally_ wrong, some people might fail to appreciate the lesson, leaving one of your students to fill in the gap in your lecture series eleven years later.
+
+The _pedagogical_ function of telling people to ["stop playing nitwit games and admit that dolphins don't belong on the fish list"](https://www.lesswrong.com/posts/d5NyJ2Lf6N22AD9PB/where-to-draw-the-boundary) is to point out that, without _denying_ the obvious similarities that motivated the initial categorization `{salmon, guppies, sharks, dolphins, trout, ...}`, there is _more structure_ in the world: to maximize the [(logarithm of the)](http://yudkowsky.net/rational/technical/) probability your world-model assigns to your observations of dolphins, you need to take into consideration the many aspects of reality in which the grouping `{monkeys, squirrels, dolphins, horses ...}` makes more sense. To the extent that relying on the initial category guess would result in a worse Bayes-score, we might say that that category is "wrong." It might have been "good enough" for the purposes of the sailors of yore, but as humanity has learned more, as our model of Thingspace has expanded with more dimensions and more details, we can see the ways in which the original map failed to carve reality at the joints.
+
+-------
+
+The one replies:
+
+> But reality doesn't come with its joints pre-labeled. Questions about how to draw category boundaries are best understood as questions about values or priorities rather than about the actual content of the actual world. I can call dolphins "fish" and go on to make just as accurate predictions about dolphins as you can. Everything we identify as a joint is only a joint because we care about it.
+
+No. Everything we identify as a joint is a joint not "because we care about it", but because it _helps us think about_ the things we care about.
+
+_Which_ dimensions of Thingspace you bother paying attention to might depend on your values, and the clusters returned by your brain's [similarity-detection](https://www.lesswrong.com/posts/jMTbQj9XB5ah2maup/similarity-clusters) algorithms might "split" or "collapse" according to which subspace you're looking at. But in order for your map to be _useful_ in the service of your values, it needs to reflect the statistical structure of things in the territory—which depends on the territory, not your values.
+
+There is an _important difference_ between "not including mountains on a map because it's a political map that doesn't show any mountains" and "not including Mt. Everest on a geographic map, because my sister died trying to climb Everest and seeing it on the map would make me feel sad."
+
+There is an _important difference_ between "identifying this pill as not being 'poison' allows me to [focus my uncertainty](https://www.lesswrong.com/posts/GJ4ZQm7crTzTM6xDW/focus-your-uncertainty) about what I'll observe after administering the pill to a human (even if [most possible minds](https://www.lesswrong.com/posts/tnWRXkcDi5Tw9rzXw/the-design-space-of-minds-in-general) have never seen a 'human' and would never waste cycles imagining administering the pill to one)" and "identifying this pill as not being 'poison', because if I publicly called it 'poison', then the manufacturer of the pill might sue me."
+
+There is an _important difference_ between having a utility function defined over a statistical model's _performance_ against specific real-world data (even if another mind with different values would be interested in different data), and having a utility function defined over features of _the model itself_.
+
+Remember how [appealing to the dictionary](https://www.lesswrong.com/posts/9ZooAqfh2TC9SBDvq/the-argument-from-common-usage) is irrational when the _actual_ motivation for an argument is about [whether to infer a property on the basis of category-membership](https://www.lesswrong.com/posts/4FcxgdvdQP45D6Skg/disguised-queries)? But at _least_ the dictionary has the virtue of documenting typical usage of our shared communication signals: you can at least see how "You're defecting from common usage" might _feel_ like a sensible thing to say, even if one's [true rejection](https://www.lesswrong.com/posts/TGux5Fhcd7GmTfNGC/is-that-your-true-rejection) lies elsewhere. In contrast, this motion of appealing to _personal values_ (!?!) is _so_ deranged that Yudkowsky apparently didn't even realize in 2008 that he might need to warn us against it!
+
+You _can't_ change the categories your mind _actually_ uses and still perform as well on prediction tasks—although you can change your [_verbally reported_](https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-the-teacher-s-password) categories, much as how one can verbally report "believing" in an [invisible, inaudible, flour-permeable dragon](https://www.lesswrong.com/posts/CqyJzDZWvGhhFJ7dY/belief-in-belief) in one's garage without having any false anticipations-of-experience about the garage.
+
+This may be easier to see with a [simple](https://www.lesswrong.com/posts/HnPEpu5eQWkbyAJCT/the-simple-math-of-everything) _numerical_ example.
+
+Suppose we have some entities that exist in the three-dimensional vector space ℝ³. There's one cluster of entities centered at [1, 2, 3], and we call those entities Foos, and there's another cluster of entities centered at [2, 4, 6], which we call Quuxes.
+
+The one comes and says, "Well, I'm going redefine the meaning of 'Foo' such that it also includes the things near [2, 4, 6] as well as the Foos-with-respect-to-the-old-definition, and you can't say my new definition is wrong, because if I observe [2, \_, \_] (where the underscores represent yet-unobserved variables), I'm going to categorize that entity as a Foo but still predict that the unobserved variables are 4 and 6, _so there_."
+
+But if the one were _actually using_ the new concept of Foo _internally_ and not just _saying the words_ "categorize it as a Foo", they _wouldn't_ predict 4 and 6! They'd predict 3 and 4.5, because those are the average values of a generic Foo-with-respect-to-the-new-definition in the 2nd and 3rd coordinates (because (2+4)/2 = 6/2 = 3 and (3+6)/2 = 9/2 = 4.5). (The already-observed 2 in the first coordinate isn't average, but by [conditional independence](https://www.lesswrong.com/posts/gDWvLicHhcMfGmwaK/conditional-independence-and-naive-bayes), that only affects our prediction of the other two variables _by means_ of its effect on our "prediction" of category-membership.) The cluster-structure knowledge that "entities for which x₁≈2, also tend to have x₂≈4 and x₃≈6" needs to be represented _somewhere_ in the one's mind _in order to get the right answer_. And given that that knowledge needs to be represented, it might also be useful to have a _word_ for "the things near [2, 4, 6]" in order to efficiently share that knowledge with others.
+
+Of course, there isn't going to be a _unique_ way to encode the knowledge into natural language: there's no reason the word/symbol "Foo" needs to represent "the stuff near [1, 2, 3]" rather than "both the stuff near [1, 2, 3] and also the stuff near [2, 4, 6]". And you might very well indeed want a [short word](https://www.lesswrong.com/posts/soQX8yXLbKy7cFvy8/entropy-and-short-codes) like "Foo" that encompasses both clusters, for example, if you want to contrast them to another cluster much farther away, or if you're mostly interested in x₁ and the difference between x₁≈1 and x₁≈2 doesn't seem large enough to notice.
+
+But if speakers of particular language were _already_ using "Foo" to specifically talk about the stuff near [1, 2, 3], then you can't swap in a new definition of "Foo" without _changing the truth values_ of sentences involving the word "Foo." Or rather: sentences involving Foo-with-respect-to-the-old-definition [are _different_ propositions](https://www.lesswrong.com/posts/shoMpaoZypfkXv84Y/variable-question-fallacies) from sentences involving Foo-with-respect-to-the-new-definition, even if they get written down using the same symbols in the same order.
+
+Naturally, all this becomes much more complicated as we move away from the simplest idealized examples.
+
+For example, if the points are more evenly distributed in configuration space rather than belonging to cleanly-distinguishable clusters, then essentialist "X is a Y" cognitive algorithms perform less well, and we get [Sorites paradox](https://plato.stanford.edu/entries/sorites-paradox/)-like situations, where we know _roughly_ what we mean by a word, but are confronted with real-world (not merely hypothetical) edge cases that we're not sure how to classify.
+
+Or it might not be obvious which dimensions of Thingspace are most relevant.
+
+Or there might be social or psychological forces anchoring word usages on identifiable [Schelling points](https://www.lesswrong.com/posts/yJfBzcDL9fBHJfZ6P/nash-equilibria-and-schelling-points) that are easy for different people to _agree_ upon, even at the cost of some statistical "fit."
+
+We could go on listing more such complications, where we seem to be faced with somewhat arbitrary choices about how to describe the world in language. But the fundamental thing is this: _the map is not the territory_. Arbitrariness in the map (what color should Texas be?) doesn't correspond to arbitrariness in the territory. Where the structure of human natural language doesn't fit the structure in reality—where we're not sure whether to say that a sufficiently small collection of sand "is a heap", because we don't know how to specify the positions of the individual grains of sand, or compute that the collection has a Standard Heap-ness Coefficient of 0.64—that's just a _bug_ in our human power of [vibratory telepathy](https://www.lesswrong.com/posts/SXK87NgEPszhWkvQm/mundane-magic). You can exploit the bug to confuse humans, but that doesn't change reality.
+
+Sometimes we might _wish_ that something to belonged to a category that it doesn't (with respect to the category boundaries that we would ordinarily use), so it's tempting to avert our attention from this painful reality with [appeal-to-arbitrariness](https://www.lesswrong.com/posts/wqmmv6NraYv4Xoeyj/conversation-halters) language-lawyering, selectively applying our philosophy-of-language skills to pretend that we can define a word any way we want with no consequences. ("I'm not late!—well, okay, we agree that I arrived half an hour after the scheduled start time, but whether I was _late_ depends on how you choose to draw the category boundaries of 'late', which is subjective.")
+
+For this reason it is said that [knowing about philosophy of language can hurt people](https://www.lesswrong.com/posts/AdYdLP2sRqPMoe8fb/knowing-about-biases-can-hurt-people). Those who know that words don't have intrinsic definitions, but _don't_ know (or have seemingly forgotten) about the [three or six dozen optimality criteria](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong) governing the use of words, can easily fashion themselves a Fully General Counterargument against _any_ claim of the form "X is a Y"—
+
+> Y doesn't unambiguously refer to the thing you're trying to point at. There's no Platonic essence of Y-ness: once we know any particular fact about X we want to know, there's no question left to ask. Clearly, you don't understand how words work, therefore I don't need to consider whether there are any non-ontologically-confused reasons for someone to say "X is a Y."
+
+[Isolated demands for rigor](https://slatestarcodex.com/2014/08/14/beware-isolated-demands-for-rigor/) are great for winning arguments against humans who aren't as philosophically sophisticated as you, but the evolved systems of perception and language by which humans process and communicate information about reality, _predate_ the Sequences. Every claim that X is a Y is an expression of [_cognitive work_](https://www.lesswrong.com/posts/QkX2bAkwG2EpGvNug/the-second-law-of-thermodynamics-and-engines-of-cognition) that cannot simply be dismissed just because most claimants doesn't know _how_ they work. Platonic essences are just the limiting case as the overlap between clusters in Thingspace goes to zero.
+
+You should _never_ say, "The choice of word is arbitrary; therefore I can say whatever I want"—which amounts to, "The choice of category is arbitrary, therefore I can believe whatever I want." If the choice were _really_ arbitrary, you would be satisfied with the choice being _made_ arbitrarily: by flipping a coin, or calling a random number generator. (It doesn't matter which.) Whatever criterion your brain is using to decide which word or belief you want, _is_ your non-arbitrary reason.
+
+If what you want isn't currently true in reality, maybe there's some action you could take to make it _become_ true. To search for that action, you're going to need accurate beliefs about what reality is _currently_ like. To enlist the help of others in your planning, you're going to need precise terminology to _communicate_ accurate beliefs about what reality is currently like. Even when—_especially_ when—the current reality is inconvenient.
+
+Even when [it hurts](https://www.lesswrong.com/posts/dHQkDNMhj692ayx78/avoiding-your-belief-s-real-weak-points).
+
+(Oh, and if you're actually trying to optimize other people's models of the world, rather than the world itself—you could just _lie_, rather than playing clever category-gerrymandering mind games. It would be a lot simpler!)
+
+-----
+
+[Imagine that you've had a peculiar job in a peculiar factory](https://www.lesswrong.com/posts/4FcxgdvdQP45D6Skg/disguised-queries) for a long time. After many mind-numbing years of sorting bleggs and rubes all day and enduring being trolled by Susan the Senior Sorter and her evil sense of humor, you finally work up the courage to ask Bob the Big Boss for a promotion.
+
+"Sure," Bob says. "Starting tomorrow, you're our new Vice President of Sorting!"
+
+"Wow, this is amazing," you say. "I don't know what to ask first! What will my new responsibilities be?"
+
+"Oh, your responsibilities will be the same: sort bleggs and rubes every Monday through Friday from 9 _a.m._ to 5 _p.m._"
+
+You frown. "Okay. But Vice Presidents get paid a lot, right? What will my salary be?"
+
+"Still &#36;9.50 hourly wages, just like now."
+
+You grimace. "O–_kay_. But Vice Presidents get more authority, right? Will I be someone's boss?"
+
+"No, you'll still report to Susan, just like now."
+
+You snort. "A Vice President, reporting to a mere Senior Sorter?"
+
+"Oh, no," says Bob. "Susan is _also_ getting promoted—to _Senior_ Vice President of Sorting!"
+
+You lose it. "Bob, this is _bullshit_. When you said I was getting promoted to Vice President, that created a bunch of probabilistic expectations in my mind: you made me _anticipate_ getting new challenges, more money, and more authority, and then you reveal that you're just slapping an inflated title on the same old dead-end job. It's like handing me a blegg, and then saying that it's a rube that just happens to be blue, furry, and egg-shaped ... or telling me you have a dragon in your garage, except that it's an invisible, silent dragon that doesn't breathe. You may _think_ you're being kind to me asking me to believe in an unfalsifiable promotion, but when you [replace the symbol with the substance](https://www.lesswrong.com/posts/GKfPL6LQFgB49FEnv/replace-the-symbol-with-the-substance), it's actually just cruel. _Stop fucking with my head!_ ... sir."
+
+Bob looks offended. "This promotion isn't _unfalsifiable_," he says. "It _says_, 'Vice President of Sorting' right here on the employee roster. That's an sensory experience that you can make falsifiable predictions about. I'll even get you business cards that say, 'Vice President of Sorting.' That's another falsifiable prediction. Using language in a way _you_ dislike is not lying. The propositions you claim false—about new job tasks, increased pay and authority—is not what the title is meant to convey, and this is known to everyone involved; it is not a secret."
+
+-------
+
+Bob _kind of_ has a point. It's tempting to argue that things like titles and names are part of the map, not the territory. Unless the name is written down. Or spoken aloud (instantiated in sound waves). Or _thought about_ (instantiated in neurons). The map is _part_ of the territory: insisting that the title isn't part of the "job" and therefore violates the maxim that meaningful beliefs must have testable consequences, doesn't quite work. Observing the title on the employee roster indeed tightly constrains your anticipated experience of the title on the business card. So, that's a non-gerrymandered, predictively useful category ... right? What is there for a rationalist to complain about?
+
+To see the problem, we must turn to information theory.
+
+Let's imagine that an abstract Job has four binary properties that can either be `high` or `low`—task complexity, pay, authority, and prestige of title—forming a four-dimensional Jobspace. Suppose that two-thirds of Jobs have `{complexity: low, pay: low, authority: low, title: low}` (which we'll write more briefly as [low, low, low, low]) and the remaining one-third have `{complexity: high, pay: high, authority: high, title: high}` (which we'll write as [high, high, high, high]).
+
+Task variety and authority are hard to perceive outside of the company, and pay is only negotiated after an offer is made, so people deciding to seek a Job can only make decisions based the Job's title: but that's fine, because in the scenario described, you can infer any of the other properties from the title with certainty. Because the properties are either _all_ low or _all_ high, the [joint entropy](https://en.wikipedia.org/wiki/Joint_entropy) of title and any other property is going to have the same value as either of the individual property entropies, namely ⅔ log₂ 3/2 + ⅓ log₂ 3 ≈ 0.918 bits.
+
+But since H(pay) = H(title) = H(pay, title), then the [mutual information](https://www.lesswrong.com/posts/yLcuygFfMfrfK8KjF/mutual-information-and-density-in-thingspace) I(pay; title) has the same value, because I(pay; title) = H(pay) + H(title) − H(pay, title) by definition.
+
+Then suppose a _lot_ of companies get Bob's bright idea: half of the Jobs that used to occupy the point [low, low, low, low] in Jobspace, get their title coordinate changed to high. So now one-third of the Jobs are at [low, low, low, low], another third are at [low, low, low, high], and the remaining third are at [high, high, high, high]. What happens to the mutual information I(pay; title)?
+
+I(pay; title) = H(pay) + H(title) − H(pay, title)  
+= (⅔ log 3/2 + ⅓ log 3) + (⅔ log 3/2 + ⅓ log 3) − 3(⅓ log 3)  
+= 4/3 log 3/2 + 2/3 log 3 − log 3 ≈ 0.2516 bits.  
+
+It went down! Bob and his analogues, having observed that employees and Job-seekers prefer Jobs with high-prestige titles, _thought_ they were being benevolent by making more Jobs have the desired titles. And perhaps they have helped [savvy employees who can arbitrage the gap between the new and old worlds](https://www.lesswrong.com/posts/8XDZjfThxDxLvKWiM/excerpts-from-a-larger-discussion-about-simulacra) by being able to put "Vice President" on their resumés when searching for a new Job.
+
+But from the perspective of people who wanted to use titles as an easily-communicable correlate of the other features of a Job, all that's actually been accomplished is _making language less useful_.
+
+-------
+
+In view of the preceding discussion, to ["37 Ways That Words Can Be Wrong"](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong), we might wish to append, "38. Your definition draws a boundary around a cluster in an inappropriately 'thin' subspace of Thingspace that excludes relevant variables, resulting in [fallacies of compression](https://www.lesswrong.com/posts/y5MxoeacRKKM3KQth/fallacies-of-compression)."
+
+Miyamoto Musashi [is quoted](http://yudkowsky.net/rational/virtues/):
+
+> The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him.
+
+Similarly, the primary thing when you take a word in your lips is your intention to reflect the territory, whatever the means. Whenever you categorize, label, name, define, or draw boundaries, you must cut through to the correct answer in the same movement. If you think only of categorizing, labeling, naming, defining, or drawing boundaries, you will not be able actually to reflect the territory.
+
+Do not ask whether there's a rule of rationality saying that you shouldn't call dolphins fish. Ask whether dolphins are fish.
+
+And if you speak overmuch of the Way you will not attain it.
+
+_(Thanks to Alicorn, Sarah Constantin, Ben Hoffman, Zvi Mowshowitz, Jessica Taylor, and Michael Vassar for feedback.)_
diff --git a/content/2020/algorithmic-intent-a-hansonian-generalized-anti-zombie-principle.md b/content/2020/algorithmic-intent-a-hansonian-generalized-anti-zombie-principle.md
new file mode 100644
index 0000000..ff20aa9
--- /dev/null
+++ b/content/2020/algorithmic-intent-a-hansonian-generalized-anti-zombie-principle.md
@@ -0,0 +1,100 @@
+Title: Algorithmic Intent: A Hansonian Generalized Anti-Zombie Principle
+Date: 2020-07-13 23:03
+Status: published
+Category: philosophy
+Tags: rationality, philosophy of language
+Slug: algorithmic-intent-a-hansonian-generalized-anti-zombie-principle
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie)
+
+> "Why didn't you tell him the truth? Were you afraid?"
+>
+> "I'm not _afraid_. I _chose_ not to tell him, because I anticipated negative consequences if I did so."
+>
+> "What do you think 'fear' _is_, exactly?"
+
+The [Generalized Anti-Zombie Principle](https://www.lesswrong.com/posts/kYAuNJX2ecH2uFqZ9/the-generalized-anti-zombie-principle) calls for us to posit "consciousness" as casually upstream of _reports_ of phenomenological experience (even if the causal link might be complicated and we might be wrong about the details of what _consciousness_ is). If you're _already_ familiar with conscious humans, then maybe you can specifically engineer a non-conscious chatbot that imitates the surface behaviors of humans talking about their experiences, but you can't have a zombie that _just happens_ to talk about being conscious _for no reason_.
+
+A similar philosophical methodology may help us understand other mental phenomena that we cannot perceive directly, but infer from behavior. The Hansonian Generalized Anti-Zombie Principle calls for us to posit "intent" as causally upstream of optimized behavior (even if the causal link might be complicated and we might be wrong about the details of what _intent_ is). You can't have a zombie that _just happens_ to systematically select actions that result in outcomes that rank high with respect to a recognizable preference ordering _for no reason_.
+
+------
+
+It's tempting to think that consciousness isn't part of the physical universe. Seemingly, we can imagine a world _physically_ identically to our own—the same atom-configurations evolving under the same laws of physics—but with no _consciousness_, a world inhabited by [philosophical "zombies"](https://www.lesswrong.com/posts/fdEWWr8St59bXLbQr/zombies-zombies) who move and talk, but only as mere automatons, without the spark of _mind_ within.
+
+It can't actually work that way. When we _talk_ about consciousness, we do so with our merely physical lips or merely physical keyboards. The causal explanation for talk about consciousness has to _either_ exist entirely within physics (in which case anything we say about consciousness is causally unrelated to consciousness, which is absurd), _or_ there needs to be some place where the laws of physics are violated as the immaterial soul is observed to be "tugging" on the brain (which is in-principle experimentally detectable). Zombies can't exist.
+
+But if consciousness exists within physics, it should respect a certain ["locality"](https://www.lesswrong.com/posts/XDkeuJTFjM9Y2x6v6/which-basis-is-more-fundamental): if the configuration-of-matter that _is you_, is conscious, then _almost_-identical configurations should also be conscious for _almost_ the same reasons. An artificial neuron that implements the same input-output relationships as a biological one, would "play the same role" within the brain, which would continue to compute the same externally-observable behavior.
+
+We don't want to say that only externally-observable behavior matters and internal mechanisms don't matter at all, because substantively different internal mechanisms could compute the same behavior. Prosaically, [acting](https://en.wikipedia.org/wiki/Acting) exists: even the best method actors aren't really occupying the same mental state that the characters they portray would be in. In the limit, we could (pretend that we could) imagine [an incomprehensibly vast Giant Lookup Table](https://www.lesswrong.com/posts/k6EPphHiBH4WWYFCj/gazp-vs-glut) that has stored the outputs that a conscious mind would have produced in response to any input. Is such a Giant Lookup Table—an entirely static mapping of inputs to outputs—conscious? Really?
+
+But this thought experiment requires us to posit the existence of a Giant Lookup Table that _just happens_ to mimic the behavior of a conscious mind. _Why_ would that happen? Why would that _actually_ happen, in the real world? (Or the closest possible world large enough to contain the Giant Lookup Table.) "Just assume it happened by coincidence, for the sake of the thought experiment" is unsatisfying, because that kind of arbitrary miracle doesn't help us understand what kind of cognitive work the ordinary [simple concept](https://www.lesswrong.com/posts/82eMd5KLiJ5Z6rTrr/superexponential-conceptspace-and-simple-words) of _consciousness_ is doing for us. You can _assume_ that a broken and scrambled egg will spontaneously reassemble itself for the sake of a thought experiment, but the interpretation of your thought-experimental results may seem tendentious given that we have [Godlike confidence](https://www.lesswrong.com/posts/q7Me34xvSG3Wm97As/but-there-s-still-a-chance-right) that [you will never, ever see that happen in the real world](https://www.lesswrong.com/posts/zFuCxbY9E2E8HTbfZ/perpetual-motion-beliefs).
+
+The [_hard_ problem of consciousness](http://www.scholarpedia.org/article/Hard_problem_of_consciousness) is still confusing unto me—it [_seems_ impossible](https://www.lesswrong.com/posts/XzrqkhfwtiSDgKoAF/wrong-questions) that any arrangement of mere matter could add up to the ineffable _qualia_ of subjective experience. But the easier and yet clearly _somehow_ related problem of how mere matter can do information-processing—can do things like construct "models" by [using sensory data to correlate its internal state with the state of the world](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence)—seems understandable, and a lot of our ordinary _use_ of the concept of _consciousness_ necessarily deals with the "easy" problems, like how perception works or how to [interpret people's self-reports](https://en.wikipedia.org/wiki/Heterophenomenology), even if we [can't _see_ the identity](https://www.lesswrong.com/posts/KmghfjH6RgXvoKruJ/hand-vs-fingers) between the hard problem and the sum of all the easy problems. Whatever the true referent of "consciousness" is—however confused our current concept of it may be—it's going to be, among other things, the cause of our [_thinking_ that we have](https://www.lesswrong.com/posts/rQEwySCcLtdKHkrHp/righting-a-wrong-question) "consciousness."
+
+If I were to punch you in the face, I can [anticipate the experience](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences) of you reacting somehow—perhaps by saying, "Ow, that really hurt! I'm perceiving an ontologically-basic _quale_ of pain right now! I hereby commit to extract a costly revenge on you if you do that again, even at disproportionate cost to myself!" The fact that the human brain has the detailed functional structure to compute that _kind_ of response, whereas rocks and trees don't, is why we can be confident that [rocks and trees don't secretly have minds like ours](https://www.lesswrong.com/posts/f4RJtHBPvDRJcCTva/when-anthropomorphism-became-stupid).
+
+We recognize consciousness by its effects because we can only recognize _anything_ by its effects. For a much simpler example, consider the idea of _sorting_. Human alphabets aren't just a set of symbols—we also have a concept of the alphabet coming in some canonical _order_. The order of the alphabet doesn't play any role in the written language itself: you wouldn't have trouble reading books from an alternate world where the order of the Roman alphabet ran _KUWONSEZYFIJTABHQGPLCMVDXR_, but all English words were the same—but you would have trouble _finding_ the books on a shelf that wasn't sorted in the order you're used to. Sorting is useful because it lets us find things more easily: "The title I'm looking for starts with a _P_, but the book in front of me starts with a _B_; skip ahead" is faster than "look at every book until you find the one".
+
+In the days before computers, the work of sorting was always done by humans: if you want your physical bookshelf to be alphabetized, you probably don't have a lot of other options than manually handling the books yourself ("This title starts with a _Pl_; I should put it ... da da da _here_, after this title starting with _Pe_ but before its neighbor starting with _Po_"). But the _computational work_ of sorting is simple enough that we can program computers to do it and _prove [theorems](https://en.wikipedia.org/wiki/Master_theorem_(analysis_of_algorithms))_ about what is being accomplished, without getting confused about the [sacred mystery](https://www.lesswrong.com/posts/6i3zToomS86oj9bS6/mysterious-answers-to-mysterious-questions) of sorting-ness.
+
+Very different systems can perform the work of sorting, but whether it's a human tidying her bookshelf, or a [punchcard-sorting machine](https://en.wikipedia.org/wiki/IBM_card_sorter), or a modern computer sorting in RAM, it's useful to have a [short word](https://www.lesswrong.com/posts/soQX8yXLbKy7cFvy8/entropy-and-short-codes) to describe _processes_ that "take in" some list of elements, and "output" a list with the same elements ordered with respect to some criterion, for which we can know that the theorems we prove about sorting-in-general will [apply to any system](http://zackmdavis.net/blog/2012/07/an-idea-for-a-psychology-experiment/) that implements sorting. (For example, sorting processes that can [only compare two items to check which is "greater"](https://en.wikipedia.org/wiki/Comparison_sort) (as opposed to being able to [exploit more detailed prior information about the distribution of elements](https://en.wikipedia.org/wiki/Sorting_algorithm#Non-comparison_sorts)) can expect to have to perform $n \log n$ comparisons, where $n$ is the length of the list.)
+
+Someone who wasn't familiar with computers might refuse to recognize sorting algorithms as _real_ sorting, as opposed to mere ["artificial sorting"](https://www.lesswrong.com/posts/YhgjmCxcQXixStWMC/artificial-addition). After all, a human sorting her bookshelf _intends_ to put the books in order, whereas the computer is just an automaton following instructions, and doesn't intend anything at all—a zombie sorter!
+
+But this position is kind of silly, a [gerrymandered concept definition](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries). To be sure, it's true that the internal workings of the human are _very_ different from that of the computer. The human wasn't special-purpose programmed to sort and is necessarily doing a lot _more_ things. The whole modality of visual perception, whereby photons bouncing off a physical copy of _Rationality: AI to Zombies_ and absorbed by the human's retina are interpreted as evidence to construct a mental representation of the book in physical reality, whose "title" "begins" with an "R", is _much more complicated_ than just storing the bit-pattern 1010010 (the [ASCII](https://en.wikipedia.org/wiki/ASCII) code for _R_) in RAM. Nor does the computer have the subjective experience of eagerly looking forward to how much easier it will be to find books after the bookshelf is sorted. The human also probably won't perform the exact same sequence of comparisons as a computer program implementing [quicksort](https://en.wikipedia.org/wiki/Quicksort)—which _also_ won't perform the same sequence of comparisons as a _different_ program implementing [merge sort](https://en.wikipedia.org/wiki/Merge_sort). But the comparisons—the act of taking two _things_ and placing them somewhere that _depends_ on which one is "greater"—need to happen _in order to get the right answer_.
+
+The concept of "sorting into alphabetical order" may have been invented before our concept of "computers", but the [most natural concept](https://www.lesswrong.com/posts/d5NyJ2Lf6N22AD9PB/where-to-draw-the-boundary) of sorting includes computers performing quicksort, merge sort, _&c._., despite the lack of intent. We might say that intent is epiphenominal _with respect to_ sorting.
+
+But even if we can understand _sorting_ without understanding intent, intent isn't epiphenominal _to the universe_. Intent is part of [the fabric of](https://www.lesswrong.com/posts/h6fzC6wFYFxxKDm8u/the-fabric-of-real-things) [stuff that makes stuff happen](https://www.lesswrong.com/posts/NhQju3htS9W6p6wE6/stuff-that-makes-stuff-happen): there are sensory experiences that will cause you to usefully attribute _intent_ to some physical systems and not others.
+
+Specifically, whatever "intent" is—however confused our current concept of it may be—it's going to be, among other things, the cause of [optimized](https://www.lesswrong.com/posts/D7EcMhL26zFNbJ3ED/optimization) behavior. We can think of something as an optimization process if it's easier to predict its effects on the world by attributing _goals_ to it, rather than by simulating its detailed actions and internal state. ["To figure out a strange plot, look at what happens, then ask who benefits."](https://www.hpmor.com/chapter/97)
+
+Alex Flint [identifies _robustness to perturbations_ as another feature of optimizing systems](https://www.lesswrong.com/posts/znfkdCoHMANwqc2WE/the-ground-of-optimization-1). If you scrambled the books on the shelf while the human was taking a bathroom break away from sorting, when she came back she would _notice_ the rearranged books, and sort them again—that's because she _intends_ to achieve the outcome of the shelf being sorted. Sorting algorithms don't, in general, have this property: if you shuffle a subarray in memory that the operation of the algorithm assumes has already been sorted, there's nothing in the code to notice or care that the "intended" output was not achieved.
+
+Note that this is a "behaviorist", "third person" perspective: we're [not talking about some subjective feeling](https://www.lesswrong.com/posts/JDLKjYKDb5ohTAY45/bad-intent-is-a-disposition-not-a-feeling) of _intending_ something, just systems that systematically steer reality into otherwise-improbable states that rank high with respect to some preference ordering.
+
+Robin Hanson often writes about [hidden motives in everyday life](http://elephantinthebrain.com/), advancing the thesis that [the criteria that control our decisions aren't the same as](https://www.lesswrong.com/posts/i6fKszWY6gLZSX2Ey/fake-optimization-criteria) the high-minded story we tell other people, and even the story we represent to ourselves. If you take a strictly first-person perspective on _intent_, the very idea of hidden motives seems absurd—a contradiction in terms. What would it even _mean_, to intend something without being aware of it? How would you _identify_ an alleged hidden motive?
+
+The answer is that positing hidden motives can simplify our predictions of behavior. It can be easier to "look backwards" from what goals the behavior achieves, and _continues_ to achieve in the presence of novel obstacles, than to "look forwards" from a detailed model of the underlying psychological mechanisms (which are [typically unknown](https://www.lesswrong.com/posts/vNBxmcHpnozjrJnJP/no-one-knows-what-science-doesn-t-know)).
+
+Hanson and coauthor Kevin Simler discuss the example of nonhuman primates grooming each other—manually combing each other's fur to remove dirt and parasites. One might assume that the function of grooming is just what it appears to be: hygiene. But that doesn't explain why primates spend more time grooming than they need to, why they predominately groom others rather than themselves, and why the amount of time a species spends grooming is unrelated to the amount of hair it has to groom, but _is_ related to the size of social groupings. These anomalies make more sense if we posit that grooming has been optimized for social-political functions, to provide a _credible_ signal of trust.[^elephant] (The [signal has to cost something](https://en.wikipedia.org/wiki/Signalling_theory)—in this case, time—in order for it to not be profitable to fake.) The hygienic function of grooming isn't unreal—parasites do in fact get removed—but the world [_looks more confusing_](https://www.lesswrong.com/posts/5JDkW4MYXit2CquLs/your-strength-as-a-rationalist) if you assume the behavior is optimized solely for hygiene.
+
+[^elephant]: Robin Hanson and Kevin Simler, _The Elephant in the Brain: Hidden Motives in Everyday Life_, Ch. 1, "Animal Behavior"
+
+This kind of multiplicity of purposes is ubiquitous: thus, [nobody does the thing they are supposedly doing](https://www.lesswrong.com/posts/8iAJ9QsST9X9nzfFy/nobody-does-the-thing-that-they-are-supposedly-doing): [politics isn't about policy](http://www.overcomingbias.com/2008/09/politics-isnt-a.html), [school is not about learning](http://www.overcomingbias.com/2010/08/school-isnt-about-learning.html), [medicine is not about health](http://www.overcomingbias.com/2008/03/showing-that-yo.html), _&c._
+
+There are functional reasons for some of the purposes of social behavior to be covert, to conceal or misrepresent information that it wouldn't be profitable for others to know. (And covert motivations might be a more effective design from an [evolutionary perspective](https://www.lesswrong.com/posts/epZLSoNvjW53tqNj9/evolutionary-psychology) than outright lying if it's too expensive to maintain two mental representations: the real map for ourselves, and a fake map for our victims.) This is sometimes explained as, "We self-deceive in order to better deceive others," but I fear that this formulation might suggest more "central planning" [on the cognitive side of the evolutionary–cognitive boundary](https://www.lesswrong.com/posts/o8Bh82hKGpRNA2q36/the-evolutionary-cognitive-boundary) than is really necessary: "self-deception" can _arise_ from different parts of the mind working at cross-purposes.
+
+Ziz [discusses the example of a father](https://sinceriously.fyi/false-faces/) attempting to practice [nonviolent communication](https://en.wikipedia.org/wiki/Nonviolent_Communication) with his unruly teenage son: the father wants to have an honest and peaceful discussion of feelings and needs, but is afraid he'll lose control and become angry and threatening.
+
+But angry threats aren't just a _random mistake_, in the way it's a random mistake if I forget to carry the one while adding 143 + 28. Random mistakes don't serve a purpose and don't resist correction: there's no plausible reason for me to _want_ the incorrect answer 143 + 28 = 161, and if you say, "Hey, you forgot to carry the one," I'll almost certainly just say "Oops" and get it right the second time. Even if I'm more likely to make arithmetic errors when I'm tired, the errors probably won't correlate in a way that _steers the future_ in a particular direction: you can't use information about _what I want_ to make better predictions about what _specific_ errors I'll make, nor use observations of specific errors to infer what I want.
+
+In contrast, the father is likely to "lose control" and make angry threats precisely _when_ peaceful behavior _isn't getting him what he wants_. That's what anger is _designed to do_: [threaten to impose costs or withhold benefits to induce conspecifics to place more weight on the angry individual's welfare](https://www.cep.ucsb.edu/topics/anger.htm).
+
+Another example of hidden motives: _Less Wrong_ commenter Caravelle [tells a story about finding a loophole in an online game](https://www.lesswrong.com/posts/DSnamjnW7Ad8vEEKd/trivers-on-self-deception?commentId=CandwLBdJXXq7Qxet), and being _outraged_ to later be accused of cheating by the game administrators—only in retrospect remembering that, on first discovering the loophole, they had specifically _told_ their teammates not to tell the administrators. The earlier Caravelle-who-discovered-the-bug must have known that the admins wouldn't allow it (or else why instruct teammates to keep quiet about it?), but the later Caravelle-who-exploited-the-bug was able to protest with perfect sincerity that they couldn't have known.
+
+Another example: someone asks me an innocuous-as-far-as-they-know question that I don't feel like answering. Maybe we're making a cake, and I feel self-conscious about my lack of baking experience. You ask, "Why did you just add an eighth-cup of vanilla?" I initially mishear you as having said, "Did you just add ..." and reply, "Yes." It's only a moment later that I realize that _that's not what you asked_: you said "_Why_ did you ...", not "_Did_ you ...". But I don't correct myself, and you don't press the point. I am not a cognitive scientist and I don't _know_ what was really going on in my brain when I misheard you: maybe my audio processing is just slow. But it seems awfully _convenient_ for me that I momentarily misheard your question _specifically_ when I didn't want to answer it and thereby reveal that I don't know what I'm doing—almost as if the elephant in my brain bet that it could get away with pretending to mishear you, and the bet paid off.
+
+Our existing language may lack the vocabulary to adequately describe optimized behavior that comes from a mixture of overt and hidden motives. Does the father _intend_ to make angry threats? Did the gamer _intend_ to cheat? Was I only _pretending_ to mishear your question, rather than actually mishearing it? We want to say _No_—not in the same sense that someone consciously intends to sort her bookshelf. And yet it seems useful to have [short codewords](https://www.lesswrong.com/posts/soQX8yXLbKy7cFvy8/entropy-and-short-codes) to talk about the aspects of these behaviors that seem _optimized_. The Hansonian Generalized Anti-Zombie Principle says that when someone "loses control" and makes angry threats, it's not because they're a zombie that _coincidentally_ happens to do so when being nice isn't getting them what they want.
+
+As Jessica Taylor explains, when our existing language lacks the vocabulary to accommodate our expanded ontology in the wake of a new discovery, one strategy for adapting our language is to define new senses of existing words that [metaphorically extend the original meaning](https://www.lesswrong.com/posts/wR4PaDp2Knu5coeXx/metaphorical-extensions-and-conceptual-figure-ground). The statement "Ice is a form of water" might be new information to a child or a primitive AI who has already seen (liquid) water, and already seen ice, but didn't _know_ that the former turns into the latter when sufficiently cold.
+
+The word _water_ in the sentence "Ice is a form of water" has a _different_ [extensional meaning](https://www.lesswrong.com/posts/HsznWM9A7NiuGsp28/extensions-and-intensions) than the word _water_ in the sentence "Water is a liquid", but both definitions can coexist as long as we're careful to precisely [disambiguate which sense](https://www.lesswrong.com/posts/y5MxoeacRKKM3KQth/fallacies-of-compression) of the word is meant in contexts where [equivocation](https://www.lesswrong.com/posts/shoMpaoZypfkXv84Y/variable-question-fallacies) [could be deceptive](http://web.archive.org/web/20200529221511/https://slatestarcodex.com/2014/11/03/all-in-all-another-brick-in-the-motte/).
+
+We might wish to apply a similar linguistic tactic in order to be able to concisely _talk_ about cases where we think someone's behavior is _optimized to achieve goals_, but the computation that determines the behavior isn't necessarily overt or conscious.
+
+_Algorithmic_ seems like a promising candidate for a disambiguating adjective to make it clear that we're talking about _the optimization criteria implied by_ a system's inputs and outputs, rather than [what it subjectively feels like to be that system](https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-algorithm-feels-from-inside). We could then speak of an "algorithmic intent" that doesn't necessarily imply "(conscious) intent", similarly to how ice is a form of "water" despite not being "(liquid) water". We might similarly want to speak of algorithmic "honesty" (referring to [signals](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution) selected on the criterion of making receivers have more accurate beliefs), ["deception"](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception) (referring to signals selected for producing _less_ accurate beliefs), or even "fraud" (_deception_ that moves resources to the agent sending the deceptive signal).
+
+Some authors might admit the pragmatic usefulness of the metaphorical extension, but insist that the new usage be marked as "just a metaphor" with a prefix such as _pseudo-_ or [_quasi-_](https://www.lesswrong.com/posts/FT9Lkoyd5DcCoPMYQ/partial-summary-of-debate-with-benquo-and-jessicata-pt-1?commentId=coWFfoYqdeuSPpTqe#vPekZcouSruiCco3c). But I claim that broad "algorithmic" senses of "mental" words like _intent_ often are more relevant and useful for making sense of the world than the original, narrower definitions that were invented by humans in the context of dealing with other humans, because the universe _in fact_ does not revolve around humans.
+
+When a predatory [_Photuris_](https://en.wikipedia.org/wiki/Photuris) firefly [sends the mating signal of a different species](https://en.wikipedia.org/wiki/Aggressive_mimicry) of firefly in order to lure prey, I think it makes sense to straight-up call this [_deceptive_](https://en.wikipedia.org/wiki/Deception_in_animals) (rather than merely pseudo- or quasi-deceptive), even though fireflies don't have language with which to think the verbal thought, "And now I'm going to send another species's mating signal in order to lure prey ..."
+
+When a [generative adversarial network](https://en.wikipedia.org/wiki/Generative_adversarial_network) learns to produce images of [realistic human faces](https://en.wikipedia.org/wiki/StyleGAN) [or anime characters](https://www.lesswrong.com/posts/mAduS8gYGFMZbNq5E/this-waifu-does-not-exist-100-000-stylegan-and-gpt-2-samples), it would in no way aid our understanding to insist that the system isn't _really_ "learning" just because it's not a human learning the way a human would—any more than it would to insist that quicksort isn't _really_ sorting. "Using exposure to data as an input into gaining capabilities" is a perfectly adequate definition of _learning_ in this context.
+
+In a nearby possible future, when you sue a company for fraud because their advertising claimed that their product would disinfect wolf bites, but the product instead gave you cancer, we would hope that the court will not be persuaded if the company's defense-lawyer AI says, "But that advertisement was composed by filtering [GPT](https://www.lesswrong.com/tag/gpt)-5 output [for the version that increased sales the most](https://www.lesswrong.com/posts/Zvu6ZP47dMLHXMiG3/optimized-propaganda-with-bayesian-networks-comment-on)—at no point did any human form the _conscious intent_ to deceive you!"
+
+Another possible concern with this proposed language usage is that if it's socially permissible to [attribute unconscious motives to interlocutors](https://web.archive.org/web/20200619200544/https://slatestarcodex.com/2019/07/17/caution-on-bias-arguments/), [people will abuse this](https://web.archive.org/web/20200619222332/https://slatestarcodex.com/2019/07/16/against-lie-inflation/) to selectively accuse their rivals of bad intent, leading to toxic social outcomes: there's no way for negatively-valenced intent-language like "fraud" or "deception" to [stably have _denotative_ meanings](https://www.lesswrong.com/posts/N9oKuQKuf7yvCCtfq/can-crimes-be-discussed-literally) independently of questions of [who should be punished](https://www.lesswrong.com/posts/r2dTchodfqX4o5DYH/blame-games).
+
+It seems plausible to me that this concern is _correct_: in a human community of any appreciable size, if you let people [question the stories we tell about ourselves](http://unremediatedgender.space/2016/Sep/psychology-is-about-invalidating-peoples-identities/), you _are_ going to get acrimonious and not-readily-falsifiable accusations of bad intent. ("_Liar!_" "Huh? You can argue that I'm wrong, but I actually believe what I'm saying!" "Oh, maybe _consciously_, but I was accusing you of being an _algorithmic_ liar.")
+
+Unfortunately, as an aspiring epistemic rationalist, [I'm _not allowed to care_](https://www.lesswrong.com/posts/bSmgPNS6MTJsunTzS/maybe-lying-doesn-t-exist#Appeals_to_Consequences_Are_Invalid) whether some descriptions might be socially harmful for a human community to adopt; I'm _only_ allowed to care about what descriptions [shorten the length of the message](https://www.lesswrong.com/posts/f4txACqDWithRi7hs/occam-s-razor) needed to describe my observations.
diff --git a/content/2020/blogging-on-less-wrong-2020-upper-half.md b/content/2020/blogging-on-less-wrong-2020-upper-half.md
deleted file mode 100644
index d623a3a..0000000
--- a/content/2020/blogging-on-less-wrong-2020-upper-half.md
+++ /dev/null
@@ -1,13 +0,0 @@
-Title: Blogging on <em>Less Wrong</em> 2020 (Upper Half)
-Date: 2020-07-18 18:46
-Status: published
-Category: meta
-Tags: elsewhere
-Slug: blogging-on-less-wrong-2020-upper-half
-
-* ["The Heckler's Veto Is Also Subject to the Unilateralist's Curse"](https://www.lesswrong.com/posts/oREzdHpzaB2RaY2fA/the-heckler-s-veto-is-also-subject-to-the-unilateralist-s)
-* ["Zoom Technologies, Inc. vs. the Efficient Markets Hypothesis"](https://www.lesswrong.com/posts/tonKatiDTzTP8LrEk/zoom-technologies-inc-vs-the-efficient-markets-hypothesis)
-* ["Comment on 'Endogenous Epistemic Factionalization'"](https://www.lesswrong.com/posts/8cWMX6L8St8k9pPRC/comment-on-endogenous-epistemic-factionalization)
-* ["Philosophy in the Darkest Timeline: Basics of the Evolution of Meaning"](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution)
-* ["Optimized Propaganda with Bayesian Networks: Comment on 'Articulating Lay Theories Through Graphical Models'"](https://www.lesswrong.com/posts/Zvu6ZP47dMLHXMiG3/optimized-propaganda-with-bayesian-networks-comment-on)
-* ["Algorithmic Intent: A Hansonian Generalized Anti-Zombie Principle"](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie) ("upper half")
diff --git a/content/2020/comment-on-endogenous-epistemic-factionalization.md b/content/2020/comment-on-endogenous-epistemic-factionalization.md
new file mode 100644
index 0000000..1c2f868
--- /dev/null
+++ b/content/2020/comment-on-endogenous-epistemic-factionalization.md
@@ -0,0 +1,152 @@
+Title: Comment on “Endogenous Epistemic Factionalization”
+Date: 2020-05-20 11:04
+Status: published
+Category: social science
+Tags: politics, game theory, Python
+Slug: comment-on-endogenous-epistemic-factionalization
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/8cWMX6L8St8k9pPRC/comment-on-endogenous-epistemic-factionalization)
+
+In ["Endogenous Epistemic Factionalization"](https://arxiv.org/abs/1812.08131) (due in a forthcoming issue of the philosophy-of-science journal [_Synthese_](https://www.springer.com/journal/11229/)), James Owen Weatherall and Cailin O'Connor propose a possible answer to the question of why people form factions that disagree on multiple subjects.
+
+The existence of persistent disagreements is [_already_](https://www.lesswrong.com/posts/NKECtGX4RZPd7SqYp/the-modesty-argument) [kind](https://www.lesswrong.com/posts/tKa9Lebyebf6a7P2o/the-rhythm-of-disagreement) [of](https://www.lesswrong.com/posts/gTTWRkSz474o7s4Dg/principles-of-disagreement) [a puzzle](https://ppe.mercatus.org/system/files/Are_Disagreements_Honest_-_WP.pdf) from a Bayesian perspective. [There's only one](https://genius.com/They-might-be-giants-one-everything-lyrics) reality. If everyone is honestly trying to get the right answer and we can all _talk_ to each other, then we should converge on the right answer (or an answer that is [less wrong](https://tvtropes.org/pmwiki/pmwiki.php/Main/TitleDrop) given the evidence we have). The fact that we _can't do it_ is, or should be, an embarrassment to our species. And the existence of _correlated_ persistent disagreements—when not only do I say "top" when you say "bottom" even after we've gone over all the arguments for whether it is in fact the case that top or bottom, but _furthermore_, the fact that I said "top" lets you _predict_ that I'll probably say "cold" rather than "hot" even _before_ we go over the arguments for that, is an _atrocity_. (Not hyperbole. Thousands of people are dying horrible suffocation deaths because we can't figure out the optimal response to a new kind of coronavirus.)
+
+Correlations between beliefs are often attributed to ideology or [tribalism](https://slatestarcodex.com/2014/11/04/ethnic-tension-and-meaningless-arguments/): if I believe that Markets Are the Answer, I'm likely to propose Market-based solutions to all sorts of seemingly-unrelated social problems, and if I'm [loyal to the Green tribe](https://www.lesswrong.com/posts/6hfGNLf4Hg5DXqJCF/a-fable-of-science-and-politics), I'm likely to [selectively censor my thoughts in order to fit the Green party line](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting). But ideology can't explain correlated disagreements on unrelated topics that the content of the ideology is silent on, and tribalism can't explain correlated disagreements on narrow, technical topics that aren't [tribal shibboleths](https://slatestarcodex.com/2016/04/04/the-ideology-is-not-the-movement/).
+
+In this paper, Weatherall and O'Connor exhibit a toy model that proposes a simple mechanism that can explain correlated disagreement: if agents disbelieve in evidence presented by those with sufficiently dissimilar beliefs, factions emerge, even though everyone is honestly reporting their observations and updating on what they are told (to the extent that they believe it). The paper didn't seem to provide source code for the simulations it describes, so I followed along in Python. (Replication!)
+
+In each round of the model, our little Bayesian agents [choose between repeatedly performing](https://en.wikipedia.org/wiki/Multi-armed_bandit) one of two actions, A or B, that can "succeed" or "fail." A is a fair coin: it succeeds exactly half the time. _As far as our agents know_, B is _either_ slightly better or slightly worse: the per-action probability of success is either 0.5 + ɛ or 0.5 − ɛ, for some ɛ (a parameter to the simulation). But secretly, we the simulation authors know that B is better.
+
+```python
+import random
+
+ε = 0.01
+
+def b():
+    return random.random() < 0.5 + ε
+```
+
+The agents start out with a uniformly random probability that B is better. The ones who currently believe that A is better, repeatedly do A (and don't learn anything, because they already know that A is exactly a coinflip). The ones who currently believe that B is better, repeatedly do B, but keep track of and publish their results in order to help everyone figure out whether B is slightly better or slightly worse than a coinflip.
+
+```python
+class Agent:
+    ...
+
+    def experiment(self):
+        results = [b() for _ in range(self.trial_count)]
+        return results
+```
+
+If $H_{+}$ represents the hypothesis that B is better than A, and $H_{-}$ represents the hypothesis that B is worse, then Bayes's theorem says
+
+$$P(H_{+}|E) = \frac{P(E|H_{+})P(H_{+})}{P(E|H_{+})P(H_{+}) + P(E|H_{-})P(H_{-})}$$
+
+where E is the record of how many successes we got in how many times we tried action B. The likelihoods $P(E|H_{+})$ and $P(E|H_{-})$ can be calculated from the probability mass function of the [binomial distribution](https://en.wikipedia.org/wiki/Binomial_distribution), so the agents have all the information they need to update their beliefs based on experiments with B.
+
+```python
+from math import factorial
+
+def binomial(p, n, k):
+    return (
+        factorial(n) / (factorial(k) * factorial(n - k)) *
+        p**k * (1 - p)**(n - k)
+    )
+
+class Agent:
+    ...
+
+    def pure_update(self, credence, hits, trials):
+        raw_posterior_good = binomial(0.5 + ε, trials, hits) * credence
+        raw_posterior_bad = binomial(0.5 - ε, trials, hits) * (1 - credence)
+        normalizing_factor = raw_posterior_good + raw_posterior_bad
+        return raw_posterior_good / normalizing_factor
+```
+
+Except in order to study the emergence of clustering among multiple beliefs, we should actually have our agents face _multiple_ "A or B" dilemmas, representing beliefs about unrelated questions. (In each case, B will again be better, but the agents don't start out knowing that.) I chose three questions/beliefs, because that's all I can fit in a pretty 3D scatterplot.
+
+If all the agents update on the experimental results published by the agents who do B, they quickly learn that B is better for all three questions. If we make a pretty 3D scatterplot where [each dimension represents](https://www.lesswrong.com/posts/WBw8dDkAWohFjWQSk/the-cluster-structure-of-thingspace) the probability that B is better for one of the dilemmas, then the points converge over time to the [1.0, 1.0, 1.0] "corner of Truth", even though they started out uniformly distributed all over the space.
+
+![](https://i.imgur.com/E61Hp4W.png)
+
+But suppose the agents don't trust each other's reports. ("Sure, she _says_ she performed $B_2$ 50 times and observed 26 successes, but she _also_ believes that $B_1$ is better than $A_1$, which is _crazy_. Are we sure she didn't just make up those 50 trials of $B_2$?") Specifically, our agents assign a probability that a report is made-up (and therefore should not be updated on) in proportion to their distance from the reporter in our three-dimensional beliefspace, and a "mistrust factor" (a parameter to the simulation).
+
+```python
+from math import sqrt
+
+def euclidean_distance(v, w):
+    return sqrt(sum((v[i] - w[i]) ** 2 for i in range(len(v))))
+
+class Agent:
+    ...
+
+    def discount_factor(self, reporter_credences):
+        return min(
+            1, self.mistrust * euclidean_distance(self.credences, reporter_credences)
+        )
+
+    def update(self, question, hits, trials, reporter_credences):
+        discount = self.discount_factor(reporter_credences)
+        posterior = self.pure_update(self.credences[question], hits, trials)
+        self.credences[question] = (
+            discount * self.credences[question] + (1 - discount) * posterior
+        )
+```
+
+(Um, the paper itself actually uses a slightly more complicated mistrust calculation that also takes into account the agent's prior probability of the evidence, but I didn't quite understand the motivation for that, so I'm going with my version. I don't think the grand moral is affected.)
+
+Then we can simulate what happens if the distrustful agents do many rounds of experiments and talk to each other—
+
+```python
+def summarize_experiment(results):
+    return (len([r for r in results if r]), len(results))
+
+def simulation(
+    agent_count,  # number of agents
+    question_count,  # number of questions
+    round_count,  # number of rounds
+    trial_count,  # number of trials per round
+    mistrust,  # mistrust factor
+):
+    agents = [
+        Agent(
+            [random.random() for _ in range(question_count)],
+            trial_count=trial_count,
+            mistrust=mistrust,
+        )
+        for i in range(agent_count)
+    ]
+
+    for _ in range(round_count):
+        for question in range(question_count):
+            experiments = []
+            for agent in agents:
+                if agent.credences[question] >= 0.5:
+                    experiments.append(
+                        (summarize_experiment(agent.experiment()), agent.credences)
+                    )
+            for agent in agents:
+                for experiment, reporter_credences in experiments:
+                    hits, trials = experiment
+                    agent.update(
+                        question,
+                        hits,
+                        trials,
+                        reporter_credences,
+                    )
+
+    return agents
+```
+
+Depending on the exact parameters, we're likely to get a result that "looks like" this `agent_count=200, round_count=20, question_count=3, trial_count=50, mistrust=2` run—
+
+![](https://i.imgur.com/hSD2pa1.png)
+
+_Some_ of the agents (depicted in red) have successfully converged on the corner of Truth, but the others have polarized into factions that are all wrong about _something_. (The colors in the pretty 3D scatterplot are a [k-means clustering](https://en.wikipedia.org/wiki/K-means_clustering) for k := 8.) On _average_, evidence pushes our agents towards Truth—note the linearity of the blue and purple points, illustrating convergence on two out of the three problems—but agents who erroneously believe that A is better (due to some combination of a bad initial credence and unlucky experimental results that failed to reveal B's ε "edge" in the sample size allotted) can end up too far away to trust those who are gathering evidence for, and correctly converging on, the superiority of B.
+
+Our authors wrap up: 
+
+> [T]his result is especially notable because there is something reasonable about ignoring evidence generated by those you do not trust—particularly if you do not trust them on account of their past epistemic failures. It would be irresponsible for scientists to update on evidence produced by known quacks. And furthermore, there is something reasonable about deciding who is trustworthy by looking at their beliefs. From my point of view, someone who has regularly come to hold beliefs that diverge from mine looks like an unreliable source of information. In other words, the updating strategy used by our agents is defensible. But, when used on the community level, it seriously undermines the accuracy of beliefs.
+
+I think the moral here is slightly off. The _specific_ something reasonable about ignoring evidence generated by those you do not trust on account of their beliefs, is the assumption that those who have beliefs you disagree with are following [a _process_ that produces systematically misleading evidence](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception). In this model, that assumption is just _wrong_. The problem isn't that the updating strategy used by our agents is individually "defensible" (what does that mean?) but produces inaccuracy "when used on the community level" (what does that mean?); the problem is that you get the wrong answer if your degree of trust doesn't match agents' actual trustworthiness. Still, it's enlighteningly disturbing to see specifically how the "distrust those who disagree" heuristic descends into the madness of factions.
+
+[(Full source code.)](https://gist.github.com/zackmdavis/49539816ee1018e524a6f5a811b5b224)
diff --git a/content/2020/december-2019-blogging-on-less-wrong.md b/content/2020/december-2019-blogging-on-less-wrong.md
deleted file mode 100644
index 4c4e451..0000000
--- a/content/2020/december-2019-blogging-on-less-wrong.md
+++ /dev/null
@@ -1,13 +0,0 @@
-Title: December 2019 Blogging on <em>Less Wrong</em>
-Date: 2020-01-01 11:16
-Status: published
-Category: meta
-Tags: elsewhere
-Slug: december-2019-blogging-on-less-wrong
-
-* ["Free Speech and Triskaidekaphobic Calculators"](https://www.lesswrong.com/posts/yaCwW8nPQeJknbCgf/free-speech-and-triskaidekaphobic-calculators-a-reply-to)
-* ["Funk-tunul's Legacy; Or, The Legend of the Extortion War"](https://www.lesswrong.com/posts/XbXJZjwinkoQXu4db/funk-tunul-s-legacy-or-the-legend-of-the-extortion-war)
-* ["Firming Up Not-Lying Around Its Edge-Cases Is Less Broadly Useful Than One Might Initially Think"](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly)
-* ["Stupidity and Dishonesty Explain Each Other Away"](https://www.lesswrong.com/posts/y4bkJTtG3s5d6v36k/stupidity-and-dishonesty-explain-each-other-away)
-* ["Speaking Truth to Power Is a Schelling Point"](https://www.lesswrong.com/posts/tCwresAuSvk867rzH/speaking-truth-to-power-is-a-schelling-point)
-* ["Don't Double-Crux With Suicide Rock"](https://www.lesswrong.com/posts/jrLkMFd88b4FRMwC6/don-t-double-crux-with-suicide-rock) ("December")
diff --git a/content/2020/dont-double-crux-with-suicide-rock.md b/content/2020/dont-double-crux-with-suicide-rock.md
new file mode 100644
index 0000000..788611b
--- /dev/null
+++ b/content/2020/dont-double-crux-with-suicide-rock.md
@@ -0,0 +1,28 @@
+Title: Don’t Double-Crux With Suicide Rock
+Date: 2020-01-01 11:02
+Status: published
+Category: philosophy
+Tags: rationality, discourse
+Slug: dont-double-crux-with-suicide-rock
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/jrLkMFd88b4FRMwC6/don-t-double-crux-with-suicide-rock)
+
+Honest rational agents should never agree to disagree.
+
+This idea is formalized in [Aumann's agreement theorem](https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem) and its various extensions ([we can't foresee to disagree](https://www.overcomingbias.com/2007/01/we_cant_foresee.html), [uncommon priors require origin disputes](http://mason.gmu.edu/~rhanson/prior.pdf), [complexity bounds](http://www.scottaaronson.com/papers/agree-econ.pdf), _&c._), but even without the sophisticated mathematics, a basic intuition should be clear: there's only one reality. Beliefs are for mapping reality, so if we're asking the same question and we're doing everything right, we should get the same answer. Crucially, even if we haven't seen the same evidence, the very fact that you believe something [_is itself evidence_](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence) that I should take into account—and you should think the same way about my beliefs.
+
+In ["The Coin Guessing Game"](https://www.overcomingbias.com/2007/01/the_coin_guessi.html), Hal Finney gives a toy model illustrating what the process of convergence looks like in the context of a simple game about inferring the result of a coinflip. A coin is flipped, and two players get a "hint" about the result (Heads or Tails) along with an associated hint "quality" uniformly distributed between 0 and 1. Hints of quality 1 always match the actual result; hints of quality 0 are useless and might as well be another coinflip. Several "rounds" commence where players simultaneously reveal their current guess of the coinflip, incorporating both their own hint and its quality, and what they can infer about the other player's hint quality from their behavior in previous rounds. Eventually, agreement is reached. The process is somewhat alien from a human perspective (when's the last time you and an interlocutor switched sides in a debate _multiple times_ before eventually agreeing?!), but not completely so: if someone whose rationality you trusted seemed visibly unmoved by your strongest arguments, you would infer that they had strong evidence or counterarguments of their own, even if there was some reason they couldn't tell you what they knew.
+
+Honest rational agents should never agree to disagree.
+
+In ["Disagree With Suicide Rock"](http://www.overcomingbias.com/2007/01/disagree_with_s.html), Robin Hanson discusses a scenario where disagreement seems clearly justified: if you encounter a rock with words painted on it claiming that you, personally, should commit suicide according to your own values, you should feel comfortable disagreeing with the words on the rock without fear of being in violation of the Aumann theorem. The rock is probably just a rock. The words are information from whoever painted them, and maybe that person _did_ somehow know something about whether future observers of the rock should commit suicide, but the rock itself doesn't [implement the dynamic](https://www.lesswrong.com/posts/CuSTqHgeK4CMpWYTe/created-already-in-motion) of responding to new evidence.
+
+In particular, if you find yourself playing Finney's coin guessing game against a rock with the letter "H" painted on it, you should just go with your own hint: it would be incorrect to reason, "Wow, the rock is still saying Heads, even after observing my belief in several previous rounds; its hint quality must have been _very_ high."
+
+_Honest_ rational agents should never agree to disagree.
+
+Human so-called "rationalists" who are aware of this may implicitly or explicitly seek agreement with their peers. If someone whose rationality you trusted seemed visibly unmoved by your strongest arguments, you might think, "Hm, we still don't agree; I should update towards their position ..."
+
+But another possibility is that [your trust has been misplaced](https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-defense-not-even-science). Humans suffering from "[algorithmic](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception) [bad](https://sinceriously.fyi/false-faces/) [faith](http://benjaminrosshoffman.com/bad-faith-behavior-not-feeling/)" are on a continuum with Suicide Rock. What matters is the counterfactual dependence of their beliefs on states of the world, not whether they know all the right keywords ("crux" and "charitable" seem to be popular these days), nor whether they can _perform the behavior_ of "making arguments"—and _definitely not_ their subjective conscious verbal narratives.
+
+And if the so-called "rationalists" around you suffer from [_correlated_](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting) algorithmic bad faith—if you find yourself living in [a world of painted rocks](https://www.lesswrong.com/posts/WHK94zXkQm7qm7wXk/asch-s-conformity-experiment)—then it may come to pass that protecting the sanctity of your map requires you to master the technique of [lonely dissent](https://www.lesswrong.com/posts/CEGnJBHmkcwPTysb7/lonely-dissent).
diff --git a/content/2020/maybe-lying-cant-exist.md b/content/2020/maybe-lying-cant-exist.md
new file mode 100644
index 0000000..433a17d
--- /dev/null
+++ b/content/2020/maybe-lying-cant-exist.md
@@ -0,0 +1,64 @@
+Title: Maybe Lying Can’t Exist?!
+Date: 2020-08-22 17:36
+Status: published
+Category: philosophy
+Tags: rationality, honesty
+Slug: maybe-lying-cant-exist
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/YptSN8riyXJjJ8Qp8/maybe-lying-can-t-exist)
+
+How is it possible to tell the truth?
+
+I mean, sure, you can use your larynx to make sound waves in the air, or you can draw a sequence of symbols on paper, but sound waves and paper-markings can't be _true_, any more than a leaf or a rock can be "true". Why do you [think you can](https://www.lesswrong.com/posts/rQEwySCcLtdKHkrHp/righting-a-wrong-question) tell the truth?
+
+This is a pretty easy question. Words don't have intrinsic ontologically-basic meanings, but intelligent systems can _learn_ associations between a symbol and things in the world. If I say "dog" and point to a dog a bunch of times, a child who didn't already know what the word "dog" meant, would soon get the idea and learn that the sound "dog" _meant_ this-and-such kind of furry four-legged animal.
+
+As a _formal_ model of how this AI trick works, [we can study sender–receiver games](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution). Two agents, a "sender" and a "receiver", play a simple game: the sender observes one of several possible states of the world, and sends one of several possible _signals_—something that the sender can vary (like sound waves or paper-markings) in a way that the receiver can detect. The receiver observes the signal, and [makes a prediction](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences) about the state of the world. If the agents both get rewarded when the receiver's prediction matches the sender's observation, a convention evolves that assigns [common-usage](https://www.lesswrong.com/posts/9ZooAqfh2TC9SBDvq/the-argument-from-common-usage) meanings to the previously and otherwise arbitrary signals. True information is communicated; the signals become a _shared_ map that reflects the territory.
+
+This works because the sender and receiver have a common interest in getting the same, correct answer—in [coordinating](https://www.lesswrong.com/posts/9QxnfMYccz9QRgZ5z/the-costly-coordination-mechanism-of-common-knowledge) for the signals to mean something. If instead the sender got rewarded when the receiver made _bad_ predictions, then if the receiver could use some correlation between the state of the world and the sender's signals in order to make better predictions, then the sender would have an incentive to change its signaling choices to destroy that correlation. No convention evolves, no information gets transferred. This case is not a matter of a map failing to reflect the territory. Rather, there just is no map.
+
+------
+
+How is it possible to _lie_?
+
+This is ... a surprisingly less-easy question. The problem is that, in the formal framework of the sender–receiver game, the meaning of a signal is simply how it makes a receiver update its probabilities, which is determined by the conditions under which the signal is sent. If I say "dog" and four-fifths of the time I point to a dog, but one-fifth of the time I point to a tree, what should a child conclude? Does "dog" mean dog-with-probability-0.8-and-tree-with-probability-0.2, or does "dog" mean dog, and I'm just lying one time out of five? (Or does "dog" mean tree, and I'm lying four times out of five?!) Our sender–receiver game model would seem to favor the first interpretation.
+
+Signals convey information. What could make a signal, information, _deceptive_?
+
+Traditionally, _deception_ has been regarded as intentionally causing someone to have a false belief. As Bayesians and reductionists, however, we endeavor to [pry open](https://www.lesswrong.com/posts/HnS6c5Xm9p9sbm4a8/grasping-slippery-things) anthropomorphic black boxes like "intent" and "belief." As a _first attempt_ at making sense of deceptive signaling, let's generalize "causing someone to have a false belief" to "causing the receiver to update its probability distribution to be less accurate ([operationalized](https://plato.stanford.edu/entries/operationalism/) as [the logarithm of the probability it assigns to the true state](https://www.lesswrong.com/posts/afmj8TKAqH6F2QMfZ/a-technical-explanation-of-technical-explanation))", and [generalize "intentionally" to](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie) "benefiting the sender (operationalized by the rewards in the sender–receiver game)".
+
+One might ask: why require the sender to benefit in order for a signal to count as deceptive? Why isn't "made the receiver update in the wrong direction" enough?
+
+The answer is that we're seeking an account of communication that _systematically_ makes receivers update in the wrong direction—signals that we can think of as having been [_optimized for_](https://www.lesswrong.com/posts/CW6HDvodPpNe38Cry/aiming-at-the-target) making the receiver make wrong predictions, rather than accidentally happening to mislead on this particular occasion. The "rewards" in this model should be interpreted mechanistically, not necessarily mentalistically: it's _just_ that things that get "rewarded" more, happen more often. That's all—and that's enough to shape the evolution of how the system processes information. There need not be any conscious mind that "feels happy" about getting rewarded (although that would do the trick).
+
+Let's test out our proposed definition of deception on a concrete example. Consider a firefly of the fictional species _[P.](https://en.wikipedia.org/wiki/Photinus_(beetle)) rey_ exploring a new area in the forest. Suppose there are three possibilities for what this area could contain. With probability 1/3, the area contains another _P. rey_ firefly of the opposite sex, available for mating. With probability 1/6, the area contains a firefly of a different species, _[P.](https://en.wikipedia.org/wiki/Photuris) redator_, which eats _P. rey_ fireflies. With probability 1/2, the area contains nothing of interest.
+
+A potential mate in the area can flash the _P. rey_ mating signal to let the approaching _P. rey_ know it's there. Fireflies evolved their eponymous ability to emit light specifically for this kind of sexual communication—potential mates have a common interest in making their presence known to each other. Upon receiving the mating signal, the approaching _P. rey_ can eliminate the predator-here and nothing-here states, and update its what's-in-this-area probability distribution from {$\frac{1}{3}$ mate, $\frac{1}{6}$ predator, $\frac{1}{2}$ nothing} to {$1$ mate}. True information is communicated.
+
+Until "one day" (in evolutionary time), a mutant _P. redator_ [emits flashes that imitate the _P. rey_ mating signal](https://en.wikipedia.org/wiki/Aggressive_mimicry), thereby luring an approaching _P. rey_, who becomes an easy meal for the _P. redator_. This meets our criteria for deceptive signaling: the _P. rey_ receiver updates in the wrong direction (revising its probability of a _P. redator_ being present downwards from $\frac{1}{6}$ to 0, even though a _P. redator_ is in fact present), and the _P. redator_ sender benefits (becoming more likely to survive and reproduce, thereby spreading the mutant alleles that predisposed it to emit _P. rey_-mating-signal-like flashes, thereby ensuring that this scenario will _systematically_ recur in future generations, even if the first time was an accident because fireflies aren't that smart).
+
+Or rather, this meets our criteria for deceptive signaling _at first_. If the _P. rey_ population counteradapts to make correct Bayesian updates in the new world containing deceptive _P. redators_, then in the new equilibrium, seeing the mating signal causes a _P. rey_ to update its what's-in-this-area probability distribution from  {$\frac{1}{3}$ mate, $\frac{1}{6}$ predator, $\frac{1}{2}$ nothing} to {$\frac{2}{3}$ mate, $\frac{1}{3}$ predator}. But now the counteradapted _P. rey_ is _not_ updating in the wrong direction. If both mates and predators send the same signal, than the [likelihood ratio](https://arbital.greaterwrong.com/p/likelihood_ratio) between them is one; the observation doesn't favor one hypothesis more than the other.
+
+So ... is the _P. redator_'s use of the mating signal _no longer deceptive_ after it's been "priced in" to the new equilibrium? Should we stop calling the flashes the "_P. rey_ mating signal" and start calling it the "_P. rey_ mating and/or _P. redator_ prey-luring signal"? Do we agree with [the executive in _Moral Mazes_ who said](https://www.lesswrong.com/posts/45mNHCMaZgsvfDXbw/quotes-from-moral-mazes#L__Truth_and_Public_Relations), "We lie all the time, but if everyone knows that we're lying, is a lie really a lie?"
+
+Some authors are willing to bite this bullet in order to preserve our tidy formal definition of _deception_. (Don Fallis and Peter J. Lewis write: "Although we agree [...] that it _seems_ deceptive, we contend that the mating signal sent by a [predator] is not _actually_ misleading or deceptive [...] not all sneaky behavior (such as failing to reveal the _whole_ truth) counts as deception".)
+
+Personally, I don't care much about having tidy formal definitions of English words; I want to understand the general [_laws_ governing](https://www.lesswrong.com/posts/eY45uCCX7DdwJ4Jha/no-one-can-exempt-you-from-rationality-s-laws) the construction and perversion of shared maps, even if a detailed understanding requires revising or splitting some of our intuitive concepts. (Cailin O'Connor writes: "In the case of deception, though, part of the issue seems to be that we generally ground judgments of what is deceptive in terms of human behavior. It may be that there is no neat, unitary concept underlying these judgments.")
+
+Whether you choose to _describe_ it with the signal/word "deceptive", "sneaky", [_Täuschung_](https://en.wiktionary.org/wiki/T%C3%A4uschung), הונאה, 欺瞞, or something else, _something_ about _P. redator's_ signal usage has the optimizing-for-the-inaccuracy-of-shared-maps property. There is a fundamental asymmetry underlying why we want to talk about a mating signal rather than a 2/3-mating-1/3-prey-luring signal, even if the latter is a better description of the information it conveys.
+
+Brian Skyrms and Jeffrey A. Barrett have an explanation in light of the observation that our sender–receiver framework is a [sequential game](https://en.wikipedia.org/wiki/Sequential_game): _first_, the sender makes an observation (or equivalently, Nature chooses the type of sender—mate, predator, or null in the story about fireflies), _then_ the sender chooses a signal, _then_ the receiver chooses an action. We can separate out the _propositional_ content of signals from their informational content by taking the propositional meaning to be defined in the [subgame](https://en.wikipedia.org/wiki/Subgame_perfect_equilibrium) where the sender and receiver have a common interest—the branches of the game tree where the players are _trying_ to communicate.
+
+Thus, we see that deception is ["ontologically parasitic" in sense that holes are](https://plato.stanford.edu/entries/holes/#Theo). You can't have a hole without some material for it to be a hole _in_; you can't have a lie without some shared map for it to be a lie _in_. And a sufficiently deceptive map, like a sufficiently holey material, collapses into noise and dust.
+
+### Bibliography
+
+I changed the species names in the standard story about fireflies because I can never remember which of _Photuris_ and _Photinus_ is which.
+
+Fallis, Don and Lewis, Peter J., ["Toward a Formal Analysis of Deceptive Signaling"](http://philsci-archive.pitt.edu/13337/)
+
+O'Connor, Cailin, _Games in the Philosophy of Biology_, §5.5, "Deception"
+
+Skyrms, Brian, _Signals: Evolution, Learning, and Information_, Ch. 6, "Deception"
+
+Skyrms, Brian and Barrett, Jeffrey A., ["Propositional Content in Signals"](http://philsci-archive.pitt.edu/14774/)
diff --git a/content/2020/message-length.md b/content/2020/message-length.md
new file mode 100644
index 0000000..80810a7
--- /dev/null
+++ b/content/2020/message-length.md
@@ -0,0 +1,220 @@
+Title: Message Length
+Date: 2020-10-19 22:52
+Status: published
+Category: philosophy
+Tags: rationality, information theory, Rust
+Slug: message-length
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/mB95aqTSJLNR9YyjH/message-length)
+
+Someone is broadcasting a stream of bits. You don't know why. A 500-bit-long sample looks like this:
+
+```
+01100110110101011011111100001001110000100011010001101011011010000001010000001010
+10100111101000101111010100100101010010101010101000010100110101010011111111010101
+01010101011111110101011010101101111101010110110101010100000001101111100000111010
+11100000000000001111101010110101010101001010101101010101100111001100001100110101
+11111111111111111100011001011010011010101010101100000010101011101101010010110011
+11111010111101110100010101010111001111010001101101010101101011000101100000101010
+10011001101010101111...
+```
+
+The thought occurs to you to [do Science to it](http://dresdencodak.com/2008/05/02/copan/)—to ponder if there's some way you could better [predict](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences) what bits are going to come next. At first you think you can't—it's just a bunch of random bits. You can't predict it, because that's what random _means_.
+
+Or does it? True, if the sequence represented flips of a fair coin—every flip independently landing either `0` or `1` with exactly equal probability—then there would be no way you could predict what would come next: any continuation you could posit would be exactly as probable as any other.
+
+But if the sequence represented flips of a _biased_ coin—if, say, `1` came up 0.55 of the time instead of exactly 0.5—then it would be possible to predict better or worse. Your [best bet for the next bit in isolation would always be `1`](https://www.lesswrong.com/posts/msJA6B9ZjiiZxT6EZ/lawful-uncertainty), and you would more strongly anticipate sequences with slightly more `1`s than `0`s.
+
+You count 265 `1`s in the sample of 500 bits. _Given_ the hypothesis that the bits were generated by a fair coin, the number of `1`s (or [without loss of generality](https://en.wikipedia.org/wiki/Without_loss_of_generality), `0`s) would be given by the binomial distribution ${500\choose k} (0.5)^k (0.5)^{500-k}$, which [has a standard deviation of](https://en.wikipedia.org/wiki/Binomial_distribution) $\sqrt{500 \cdot 0.5^2} = \sqrt{125} \approx 11.18$, so your observation of $265 - 250 = 15$ excess `1`s is about $\frac{15}{11.18} \approx 1.34$ standard deviations from the mean—well within the realm of plausibility of happening by chance, although you're at least slightly _suspicious_ that the coin behind these bits might not be quite fair.
+
+... that is, if it's even a coin. You love talking in terms of shiny, if hypothetical, "coins" rather than stodgy old "[independent and identically distributed](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables) [binary-valued random variables](https://en.wikipedia.org/wiki/Bernoulli_distribution)", but looking at the sample again, you begin to _further_ doubt whether the bits are independent of each other. You've [heard that humans are biased](https://www.lesswrong.com/posts/6xGC9P8wp2mi7uhti/inaccessible-finely-tuned-rng-in-humans) to overestimate the frequency of alternations (`101010`...) and underestimate the frequency of consecutive runs (`00000`... or `11111`...) in "truly" (uniformly) random data, but the 500-bit sample contains a run of 13 `0`s (starting at position 243) _and_ a run of 19 `1`s (starting at position 319). You're not immediately sure how to [calculate](http://www.gregegan.net/QUARANTINE/Runs/Runs.html) the [probability](https://www.johndcook.com/blog/2012/11/14/probability-of-long-runs/) of that, but your gut says that should be very unlikely given the biased-coin model, even after taking into account that human guts aren't very good at estimating these things.
+
+Maybe not everything in the universe is a coin. What if the bits were being generated by a [Markov chain](https://en.wikipedia.org/wiki/Markov_chain)—if the probability of the next bit depended on the value of the one just before? If a `0` made the _next_ bit more likely to be a `0`, and the same for `1`, that would make the `00000`... and `11111`... runs less improbable.
+
+Except ... the sample _also_ has a run of 17 alternations (starting at position 153). On the "fair coin" model, shouldn't that itself be $2^{17-13} = 16$ times as suspicious as the run of 13 `0`s and $2^{17-19} = \frac{1}{4}$ as suspicious as the run of 19 `1`s which led you to hypothesize a Markov chain?—or rather, 8 and $\frac{1}{8}$ times as suspicious, respectively, because there are two ways for an alternation to occur (`0101010`... or `1010101`...).
+
+A Markov chain in which a `0` or `1` makes another of the same more likely, makes alternations _less_ likely: the Markov chain hypothesis can only make the consecutive runs look less surprising at the expense of making the run of alternations look _more_ surprising.
+
+So maybe it's all just a coincidence: the broadcast is random—whatever that means—and you're just apophenically pattern-matching on noise. Unless ...
+
+Could it be that some things in the universe are _neither_ coins _nor_ Markov chains? You don't _know_ who is broadcasting these bits or why; you called it "random" because [you didn't see](https://www.lesswrong.com/posts/f6ZLxEWaankRZ2Crv/probability-is-in-the-mind) any obvious pattern, but now that you think about it, it would be pretty weird for someone to just be broadcasting random bits. Probably the broadcast is something like a movie or a stock ticker; if a close-up sample of the individual bits looks "random", that's only because you don't know the [codec](https://en.wikipedia.org/wiki/Codec).
+
+Trying to guess a video codec is [_obviously_ impossible](https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien-message). Does that kill all hope of being able to better predict future bits? _Maybe_ not. Even if you don't know what the broadcast is really for, there might be some nontrivial _local_ structure to it, where bits are statistically related to the bits nearby, like how a dumb encoding of a video might have consecutive runs of the same bit-pattern where a large portion of a frame is the same color, like the sky.
+
+Local structure, where bits are statistically related to the bits nearby ... kind of like a Markov chain, except in a Markov chain the probability of the next state only depends on the _one_ immediately before, which is a pretty narrow notion of "nearby." To broaden that, you could imagine the bits are being generated by a [_higher-order_ Markov chain](https://en.wikipedia.org/wiki/Discrete-time_Markov_chain#Variations), where the probability of the next bit depends on the previous _n_ bits for some specific value of _n_.
+
+And _that's_ how you can explain mysteriously frequent consecutive runs and alternations. If the last _two_ bits being `01` (respectively `10`) makes it more likely for the next bit to be `0` (respectively `1`), _and_ the last two bits being `00` (respectively `11`) makes it more likely for the next bit to be `0` (respectively `1`), then you would be more likely to see both long `0000`... or `1111`... consecutive runs _and_ `01010`... alternations.
+
+A biased coin is just an _n_-th-order Markov chain where _n_ = 0. An _n_-th-order Markov chain where _n_ > 1, is just a first-order Markov chain where each "state" is a tuple of bits, rather than a single bit. 
+
+![](https://i.imgur.com/frm9VkC.png)
+
+Everything in the universe is a Markov chain!—with respect to the models you've considered so far.
+
+"The bits are being generated by a Markov chain of some order" is _a_ theory, but a pretty broad one. To make it concrete enough to test, you need to posit some specific order _n_, and, given _n_, specific parameters for the next-bit-given-previous-_n_ probabilities.
+
+The _n_ = 0 coin has one parameter: the bias of the coin, the probability of the next bit being `0`. (Or without loss of generality `1`; we just need one parameter _p_ to specify the probability of one of the two possibilities, and then the probability of the other will be 1 − _p_.)
+
+The _n_ = 1 ordinary Markov chain has two parameters: the probability of the next bit being (without loss of generality) `0` _given_ that the last bit was a `0`, and the probability of the next bit being (without loss of ...) `0` _given_ that the last bit was a `1`.
+
+The _n_ = 2 second-order Markov chain has four parameters: the probability of the next bit being (without loss ...) `0` _given_ that the last two bits were `00`, the probability of the next bit being (without ...) `0` _given_ that the last two bits were `01`, the probability of the next bit being—
+
+_Enough!_ You get it! The _n_-th order Markov chain has $2^{n}$ parameters!
+
+Okay, but then how do you guess the parameters?
+
+For the _n_ = 0 coin, your _best guess_ at the frequency-of-`0` parameter is going to just be the frequency of `0`s you've observed. Your best guess could easily be wrong, and probably is: just because you observed 235/500 = 0.47 `0`s, doesn't mean the parameter _is_ 0.47: it's probably somewhat lower or higher, and your sample got more or fewer `0`s than average just by chance. But positing that the observed frequency is the actual parameter is the [maximum likelihood estimate](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation)—the single value that _most_ makes the data ["look normal"](https://www.lesswrong.com/posts/tWLFWAndSZSYN6rPB/think-like-reality).
+
+For _n_ ≥ 1, it's the same idea: your best guess for the frequency-of-`0`-after-`0` parameter is just the frequency of `0` being the next bit, among all the places where `0` was the last bit, and so on.
+
+You can write a program that takes the data and a degree _n_, and computes the maximum-likelihood estimate for the _n_-th order Markov chain that might have produced that data. Just slide an (_n_+1)-bit "window" over the data, and keep a tally of the frequencies of the "plus-one" last bit, for each of the $2^{n}$ possible _n_-bit patterns.
+
+In the [Rust](https://www.rust-lang.org/) programming language, that looks like following. (Where the representation of our final theory is output as a [map](https://en.wikipedia.org/wiki/Hash_table) (`HashMap`) from _n_+1-bit-patterns to frequencies/parameter-values (stored as a thirty-two bit floating-point number, `f32`).)
+
+```rust
+fn maximum_likelihood_estimate(data: &[Bit], degree: usize) -> HashMap<(Vec<Bit>, Bit), f32> {
+    let mut theory = HashMap::with_capacity(2usize.pow(degree as u32));
+    // Cartesian product—e.g., if degree 2, [00, 01, 10, 11]
+    let patterns = bit_product(degree);
+    for pattern in patterns {
+        let mut zero_continuations = 0;
+        let mut one_continuations = 0;
+        for window in data.windows(degree + 1) {
+            let (prefix, tail) = window.split_at(degree);
+            let next = tail[0];
+            if prefix == pattern {
+                match next {
+                    ZERO => {
+                        zero_continuations += 1;
+                    }
+                    ONE => {
+                        one_continuations += 1;
+                    }
+                }
+            }
+        }
+        let continuations = zero_continuations + one_continuations;
+        theory.insert(
+            (pattern.clone(), ZERO),
+            zero_continuations as f32 / continuations as f32,
+        );
+        theory.insert(
+            (pattern.clone(), ONE),
+            one_continuations as f32 / continuations as f32,
+        );
+    }
+    theory
+}
+```
+
+Now that you have the best theory for each particular _n_, you can compare how well each of them predict the data! For example, according to _n_ = 0 coin model with maximum-likelihood parameter _p_ = 0.47, the probability of your 500-bit sample is about ... 0.0000000000000000000000000000000000000000000000000000000000000000000000000
+000000000000000000000000000000000000000000000000000000000000000000000000000  
+007517883433770135.
+
+Uh. The tiny probability makes sense: there's a _lot_ of randomness in 500 flips of a biased coin. Even if you know the bias, the probability of any _particular_ 500-flip sequence is going to be tiny. But a number that tiny is kind of unwieldy to work with. You'd almost rather just count the zeros and ignore the specific digits afterwards.
+
+But counting the zeros is just taking the logarithm—well, the negative logarithm in the case of zeros after the decimal point. Better make the log base-two—it's _thematic_. Call this measurement the _log loss_.
+
+```rust
+fn log_loss(theory: &HashMap<(Vec<Bit>, Bit), f32>, data: &[Bit]) -> f32 {
+    let mut total = 0.;
+    let degree = log2(theory.keys().count()) - 1;
+    for window in data.windows(degree + 1) {
+        let (prefix, tail) = window.split_at(degree);
+        let next = tail[0];
+        total += -theory
+            .get(&(prefix.to_vec(), next))
+            .expect("theory should have param value for prefix-and-continuation")
+            .log2();
+    }
+    total
+}
+```
+
+_Now_ you can compare different theories to see which order of Markov chain is _the best_ theory to "fit" your 500-bit sample ... right?
+
+```rust
+    for hypothesized_degree in 0..15 {
+        let theory = maximum_likelihood_estimate(&data, hypothesized_degree);
+        println!(
+            "{}th-order theory: fit = {}",
+            hypothesized_degree,
+            log_loss(&theory, &data)
+        );
+    }
+```
+
+```
+0th-order theory: fit = 498.69882
+1th-order theory: fit = 483.86075
+2th-order theory: fit = 459.01752
+3th-order theory: fit = 438.90198
+4th-order theory: fit = 435.9401
+5th-order theory: fit = 425.77222
+6th-order theory: fit = 404.2693
+7th-order theory: fit = 344.68494
+8th-order theory: fit = 270.51175
+9th-order theory: fit = 199.88765
+10th-order theory: fit = 147.10117
+11th-order theory: fit = 107.72962
+12th-order theory: fit = 79.99724
+13th-order theory: fit = 57.16126
+14th-order theory: fit = 33.409912
+```
+
+There's a problem. Higher choices of _n_ monotonically achieve a better "fit". You got the idea of higher-order Markov chains because the idea of a biased coin didn't seem adequate to explain the consecutive and alternating runs you saw, but you somehow have trouble believing that the bitstream was generated by a _fifteenth_-order Markov chain with a completely separate probability for the next bit for _each_ of the $2^{15}$ = 32,768 prefixes `000000000000000`, `000000000000001`, `000000000000010`, _&c._ Having had the "higher-order Markov chain" idea, are you now obligated to set _n_ as large as possible? What would that even _mean_?
+
+In retrospect, the problem should have been obvious from the start. Using your sample data to choose maximum-likelihood parameters, and then using the model with those parameters to "predict" the _same_ data puts you in the position of [the vaunted "sharpshooter"](https://en.wikipedia.org/wiki/Texas_sharpshooter_fallacy) who paints a target around a clump of bullet holes _after_ firing wildly at the broad side of a barn.
+
+Higher values of _n_ [are like](https://en.wikipedia.org/wiki/Overfitting) a ... thinner paintbrush?—or a squigglier, more "gerrymandered" painting of a target. Higher-order Markov chains are _strictly_ more expressive than lower-order ones: the zeroth-order coin is just a first-order Markov chain where the next-bit-after-`0` and next-bit-after-`1` parameters just happen to be the same; the first-order Markov chain is just a second-order chain where the next-bit-after-`00` and next-bit-after-`10` parameters happen to be the same, as well as the next-bit-after-`01` and—_enough!_ You get it!
+
+The broadcast is ongoing; you're not limited to the particular 500-bit sample you've been playing with. If the worry were _just_ that the higher-order models will (somehow, you intuit) fail to predict future data, you could [use different samples for estimating parameters and validating the resulting models](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets), but you think you're suffering from some more fundamental confusion—one that's probably not limited to Markov chains in particular.
+
+Your working concept of what it means for a theory to "fit" the data, is for it to maximize the probability with which the theory predicts the data. This is an objective, _quantitative_ measurement. (Okay, the log loss is taking the negative logarithm of that to avoid so many zeros after the decimal point, but minimizing the log loss and maximizing the probability are both expressing the same preference on theories.)
+
+_How_ do you know (and your gut says that you _know_) that the higher-order models will do badly on future data, if your objective criterion of model-goodness says they're better? The log loss always "wants" to you to choose ever-more-complex models. You asked: what would that even _mean_? But maybe it doesn't have to be a rhetorical question: what _would_ that even mean?
+
+Well ... in the limit, you could choose a theory that assigns [Probability One](https://www.lesswrong.com/posts/ooypcn7qFzsMcy53R/infinite-certainty) to the observed data. The "too many zeros"/"avoid working with really tiny numbers" justification for taking the negative log doesn't really apply here, but for consistency with your earlier results, you dutifully note that the logarithm of 1 is 0 ...
+
+But maybe "too many zeros" isn't the only good motivation for taking the logarithm? [_Intelligence is prediction is compression_.](https://www.lesswrong.com/posts/ex63DPisEjomutkCw/msg-len) The log loss of a model against the data can be interpreted as the [expected number of bits](https://arbital.greaterwrong.com/p/fractional_bits_as_expected_cost) you would need to describe the data, given the [optimal code](https://www.lesswrong.com/posts/soQX8yXLbKy7cFvy8/entropy-and-short-codes) implied by your model.
+
+In order to communicate a reduction in your uncertainty, [you need to send a signal](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution)—something you can choose to vary in response to the reality of the data. A signal you can vary to take two possible states, can distinguish between two sets among which you've divided the remaining possibilities; writing down a bit means halving your uncertainty.
+
+On this interpretation, what the logarithm of Probability One being zero _means_ is that if your theory predicted the exact outcome with certainty, then once you stated the theory, you wouldn't have to say anything more in order to describe the data—you would just _know_ with zero further bits.
+
+_Once you stated the theory_. A theory implies an optimally efficient coding by which further bits can whittle down the space of possibilities to the data that actually happened. More complicated or unlikely data requires more bits just to _specify_—to single out that one outcome amongst the vastness of equally remote alternatives. But [_the same thing goes for theories_](https://www.lesswrong.com/posts/nj8JKFoLSMEmD3RGp/how-much-evidence-does-it-take).
+
+Given a particular precision to which parameters are specified, there are exponentially more Markov chains of higher degrees, which can continue to drive down the log loss—but not faster than their _own_ probability decreases. You need exponentially more data just to learn the parameters of a higher-order model. [If you don't _have_ that much data](https://www.lesswrong.com/posts/X2AD2LgtKgkRNPj2a/privileging-the-hypothesis)—enough to pin down the $2^n$ parameters that single out this _particular_ higher-order Markov chain amongst the vastness of equally remote alternatives—then your maximum-likelihood best guess is not going to be very good on future data, for the same reason you [can't expect to correctly guess](https://www.lesswrong.com/posts/zFuCxbY9E2E8HTbfZ/perpetual-motion-beliefs) that a biased coin has a probability of landing Heads of exactly 0.23 if you've only seen it flipped twice.
+
+If you _do_ have enough data to learn a more complex model, but the data was actually generated by a simpler model, then the parameters of the complex model will approximately take the settings that produce the same behavior as the simpler model—like a second-order Markov chain for which the bit-following-`01` parameter happens to take the same value as the bit-following-`11` parameter. And if you're deciding what theory to prefer based on both fit and complexity, [the more complex model won't be able to "pay" for its increased complexity](https://www.lesswrong.com/posts/f4txACqDWithRi7hs/occam-s-razor) with its own predictions.
+
+Now that you know what's going on, you can [modify your code to penalize](https://www.lesswrong.com/posts/H59YqogX94z5jb8xx/inductive-bias) more complex models. Since the parameters in your implementation are 32-bit floats, you assign a complexity cost of $32 \cdot 2^n$ bits to _n_-th order Markov chains, and look at the sum of fit (log loss) and complexity. Trying out your code again on a larger sample of 10,000 bits from the broadcast—
+
+```rust
+    for hypothesized_degree in 0..10 {
+        let theory = maximum_likelihood_estimate(&data, hypothesized_degree);
+        let fit = log_loss(&theory, &data);
+        let complexity = 2f32.powi(hypothesized_degree as i32) * 32.;
+        println!(
+            "{}th-order theory: fit = {}, complexity = {}, total = {}",
+            hypothesized_degree, fit, complexity, fit + complexity
+        );
+    }
+```
+
+```
+0th-order theory: fit = 9970.838, complexity = 32, total = 10002.838
+1th-order theory: fit = 9677.269, complexity = 64, total = 9741.269
+2th-order theory: fit = 9111.029, complexity = 128, total = 9239.029
+3th-order theory: fit = 8646.953, complexity = 256, total = 8902.953
+4th-order theory: fit = 8638.786, complexity = 512, total = 9150.786
+5th-order theory: fit = 8627.224, complexity = 1024, total = 9651.224
+6th-order theory: fit = 8610.54, complexity = 2048, total = 10658.54
+7th-order theory: fit = 8562.568, complexity = 4096, total = 12658.568
+8th-order theory: fit = 8470.953, complexity = 8192, total = 16662.953
+9th-order theory: fit = 8262.546, complexity = 16384, total = 24646.547
+```
+
+—reveals a clear preference for the third-order theory (that for which the fit-plus-complexity score is the lowest), allowing you to enjoy the huge 450-plus–bit leap in compression/prediction from _n_ := 2 to 3 and _logically stop there_, the steepness of the ascent into the madness of arbitrary complexity successfully dissuading you from chasing after diminishing returns (which [you suspect](https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theorem) are only hallucinatory). That's the power packed by parsimony—the sublime simplicity of _Science_.
+
+[(Full source code.)](https://gist.github.com/zackmdavis/e89afcd47de746fa8b510f3cf2733203)
diff --git a/content/2020/msg-len.md b/content/2020/msg-len.md
new file mode 100644
index 0000000..4430cc2
--- /dev/null
+++ b/content/2020/msg-len.md
@@ -0,0 +1,15 @@
+Title: Msg Len
+Date: 2020-10-11 20:35
+Status: published
+Category: verse
+Tags: poetry, information theory
+Slug: msg-len
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/ex63DPisEjomutkCw/msg-len)
+
+I'll be brief, omit needless words.  
+Intelligence is prediction is compression _because_  
+Compression is finding a code that makes the data shorter  
+And codeword lengths are probabilities  
+So codes are probability distributions  
+But probability distributions are prediction strategies.
diff --git a/content/2020/optimized-propaganda-with-bayesian-networks-comment-on-articulating-lay-theories-through-graphical-models.md b/content/2020/optimized-propaganda-with-bayesian-networks-comment-on-articulating-lay-theories-through-graphical-models.md
new file mode 100644
index 0000000..298e266
--- /dev/null
+++ b/content/2020/optimized-propaganda-with-bayesian-networks-comment-on-articulating-lay-theories-through-graphical-models.md
@@ -0,0 +1,56 @@
+Title: Optimized Propaganda with Bayesian Networks: Comment on “Articulating Lay Theories Through Graphical Models”
+Date: 2020-06-28 19:45
+Status: published
+Category: social science
+Tags: Bayes-structure of the universe, politics
+Slug: optimized-propaganda-with-bayesian-networks-comment-on-articulating-lay-theories-through-graphical-models
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/Zvu6ZP47dMLHXMiG3/optimized-propaganda-with-bayesian-networks-comment-on)
+
+Derek Powell, Kara Weisman, and Ellen M. Markman's ["Articulating Lay Theories Through Graphical Models: A Study of Beliefs Surrounding Vaccination Decisions"](http://www.derekmpowell.com/publication/lay-theories-cogsci) (a conference paper from [CogSci 2018](https://cognitivesciencesociety.org/past-conferences/)) represents an exciting advance in marketing research, showing how to use [causal graphical models](https://www.lesswrong.com/posts/hzuSDMx7pd2uxFc5w/causal-diagrams-and-causal-models) to study why ordinary people have the beliefs they do, and how to intervene to make them be [less wrong](https://tvtropes.org/pmwiki/pmwiki.php/Main/TitleDrop).
+
+The specific case our authors examine is that of childhood vaccination decisions: some parents don't give their babies the recommended vaccines, because they're afraid that vaccines cause autism. [(Not true.)](https://en.wikipedia.org/wiki/MMR_vaccine_and_autism) This is pretty bad—not only are those unvaccinated kids more likely to get sick themselves, but declining vaccination rates undermine the population's [herd immunity](https://en.wikipedia.org/wiki/Herd_immunity), leading to [new outbreaks of highly-contagious diseases like the measles in regions where they were once eradicated](https://en.wikipedia.org/wiki/Measles_resurgence_in_the_United_States).
+
+What's wrong with these parents, huh?! But that doesn't have to just be a rhetorical question—Powell _et al._ show how we can use statistics to make the rhetorical [hypophorical](https://en.wikipedia.org/wiki/Hypophora) and model _specifically_ what's wrong with these people! Realistically, people aren't going to just have a raw, "atomic" dislike of vaccination _for no reason_: parents who refuse to vaccinate their children do so _because_ they're (irrationally) afraid of giving their kids autism, and not afraid enough of letting their kids get infectious diseases. Nor are beliefs about vaccine effectiveness or side-effects _uncaused_, but instead depend on other beliefs.
+
+To unravel the structure of the web of beliefs, our authors got [Amazon Mechanical Turk](https://en.wikipedia.org/wiki/Amazon_Mechanical_Turk) participants to take surveys about vaccination-related beliefs, rating statements like "Natural things are always better than synthetic alternatives" or "Parents should trust a doctor's advice even if it goes against their intuitions" on a 7-point [Likert-like scale](https://en.wikipedia.org/wiki/Likert_scale) from "Strongly Agree" to "Strongly Disagree".
+
+Throwing some [off-the-shelf Bayes-net structure-learning software](https://www.bnlearn.com/) at a [training set](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets) from the survey data, plus some ancillary assumptions (more-general "theory" beliefs like "skepticism of medical authorities" can cause more-specific "claim" beliefs like "vaccines have harmful additives", but not _vice versa_) produces a range of probabilistic models that can be depicted with [graphs](https://en.wikipedia.org/wiki/Graph_(discrete_mathematics)) where nodes representing the different beliefs are connected by arrows that show which beliefs "cause" others: an arrow from a _naturalism_ node (in this context, denoting a worldview that prefers natural over synthetic things) to a _parental expertise_ node means that people think parents know best _because_ they think that nature is good, not the other way around.
+
+Learning [these kinds of models](https://www.lesswrong.com/posts/jnjjzkH8Fdzg4D6EK/causality-a-chapter-by-chapter-review) is feasible because not all possible causal relationships are consistent with the data: if $A$ and $B$ are [statistically independent](https://en.wikipedia.org/wiki/Independence_(probability_theory)) of each other, but each dependent with $C$ (and are [_conditionally_](https://en.wikipedia.org/wiki/Conditional_independence) dependent given the value of $C$), it's kind of hard to make sense of this except to posit that $A$ and $B$ are causes with the common effect $C$.
+
+Simpler models with fewer arrows might sacrifice a little bit of predictive accuracy for the benefit of being more intelligible to humans. Powell _et al._ ended up choosing a model that can predict responses from the [test set](https://en.wikipedia.org/wiki/Cross-validation_(statistics)) at [_r_](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) = .825, [explaining](https://en.wikipedia.org/wiki/Explained_variation) 68.1% of the variance. Not bad?!—check out the full 14-node graph in Figure 2 on page 4 of [the PDF](https://mindmodeling.org/cogsci2018/papers/0183/0183.pdf).
+
+Causal graphs are useful as a guide for planning interventions: the graph encodes predictions about what would happen if you _changed_ some of the variables. Our authors point out that since [previous work](https://www.pnas.org/content/112/33/10321) showed that people's beliefs about vaccine dangers were difficult to influence, that suggests trying to intervene on the _other_ parents of the intent-to-vaccinate node in the model: if the _hoi polloi_ won't listen to you when you tell them the costs are minimal (vaccines are safe), instead tell them about the benefits (diseases are really bad and vaccines prevent disease).
+
+To make sure I really understand this, I want to adapt it into a simpler example with made-up numbers where I can do the arithmetic myself. Let me consider a graph with just three nodes—
+
+![vaccines are safe → vaccinate against measles ← measles are dangerous](https://i.imgur.com/NuYrnik.png)
+
+Suppose this represents a [structural equation model](https://en.wikipedia.org/wiki/Structural_equation_modeling) where an anti-vaxxer-leaning parent-to-be's propensity-to-vaccinate-against-measles $C$ is expressed in terms of belief-in-vaccine-safety $A$ and belief-in-measles-danger $B$ as—
+
+$$C = 0.7 \cdot A + 0.3 \cdot B $$
+
+And suppose that we're a public health authority trying to decide whether to spend our budget (or what's left of it after recent funding cuts) on a public education initiative that will increase $A$ by 0.1, or one that will increase $B$ by 0.3.
+
+We should choose the program that intervenes on $B$, because $(0.3)(0.3) = 0.09$ is bigger than $(0.7)(0.1) = 0.07$. That's actionable advice that we couldn't have derived without a quantitative model of how the lay audience thinks. Exciting!
+
+At this point, some readers may be wondering why I've described this work as "marketing research" about constructing "optimized propaganda." A couple of those words usually have _negative_ connotations, but educating people about the importance of vaccines is a _positive_ thing. What gives?
+
+The thing is, "Learn the causal graph of why they think that and compute how to intervene on it to make them think something else" is a [symmetric weapon](https://web.archive.org/web/20200521005958/http://slatestarcodex.com/2017/03/24/guided-by-the-beauty-of-our-weapons/)—a _fully general_ persuasive technique that doesn't [depend on whether the thing you're trying to convince them of is _true_](http://benjaminrosshoffman.com/humility-argument-honesty/).
+
+In my simplified example, the choice to intervene on $B$ was based on numerical assumptions that amount to the claim that it's sufficiently easier to change $B$ than it is to change $A$, such that intervening on $B$ is more effective at changing $C$ than intervening on $A$ (even though $C$ depends on $A$ more than it does on $B$). But this methodology is _completely indifferent_ to what $A$, $B$, and $C$ _mean_. It would have worked just as well, and _for the same reasons_ if the graph had been—
+
+![Coca-Cola isn't unhealthy → drink Coca-Cola ← Coca-Cola tastes great](https://i.imgur.com/lQmo66J.png)
+
+Suppose that we're advertising executives for the Coca-Cola Company trying to decide how to spend our budget (or what's left of it after recent funding cuts). If consumers won't listen to us when we tell them the costs of drinking Coke are minimal (lying that it isn't unhealthy), we should instead tell them about the benefits (Coke tastes good).
+
+Or with different assumptions about the parameters—maybe $C = 0.8 \cdot A + 0.2 \cdot B$ actually—then intervening to increase belief in "Coca-Cola isn't unhealthy" _would_ be the right move (because $(0.8)(0.1) = 0.08 > 0.06 = (0.2)(0.3)$). The [marketing algorithm](https://www.lesswrong.com/posts/P3FQNvnW8Cz42QBuA/dialogue-on-appeals-to-consequences?commentId=bAQBHN2svqS6BfmSM) that just computes _what belief changes will flip the decision node_, doesn't have any way to notice or care whether those belief changes are in the direction of more or less accuracy.
+
+To be clear—and I really _shouldn't_ have to say this—this is not a criticism of Powell–Weisman–Markman's research! The "Learn the causal graph of why they think that" methodology is genuinely really cool! It doesn't _have_ to be deployed as a marketing algorithm: the process of figuring out which belief change would flip some downstream node is the same thing as what we call locating a [crux](https://www.lesswrong.com/posts/exa5kmvopeRyfJgCy/double-crux-a-strategy-for-resolving-disagreement).[^crux] The difference is just a matter of [forwards or backwards direction](https://www.lesswrong.com/posts/SFZoEBpLo9frSJGkc/rationalization): whether you _first_ figure out if the measles vaccine or Coca-Cola are safe and [_then_ use whatever answer you come up with to guide your decision](https://www.lesswrong.com/posts/9f5EXt8KNNxTAihtZ/a-rational-argument), or whether you [write the bottom line first](https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-line).
+
+[^crux]: Thanks to [Anna Salamon](https://www.lesswrong.com/users/annasalamon) for this observation.
+
+Of course, most people on most issues don't have the time or expertise to do their own research. For the most part, we can only hope that the sources we trust as authorities are doing their best to use their [limited bandwidth](https://www.lesswrong.com/posts/4ZvJab25tDebB8FGE/you-have-about-five-words) to keep us genuinely informed, rather than merely computing what [signals to emit](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution) in order to control our decisions.
+
+If that's _not_ true, we might be in trouble—perhaps increasingly so, if technological developments grant new advantages to the propagation of disinformation over the discernment of truth. In [a possible future world](https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/what-failure-looks-like) where _most_ words are produced by AIs running a "Learn the causal graph of why they think that and intervene on it to make them think something else" algorithm hooked up to a next-generation [GPT](https://www.lesswrong.com/tag/gpt), even [reading plain text from an untrusted source could be dangerous](https://www.alignmentforum.org/posts/5bd75cc58225bf06703754b9/autopoietic-systems-and-difficulty-of-agi-alignment?commentId=5bd75cc58225bf06703754c1).
diff --git a/content/2020/philosophy-in-the-darkest-timeline-basics-of-the-evolution-of-meaning.md b/content/2020/philosophy-in-the-darkest-timeline-basics-of-the-evolution-of-meaning.md
new file mode 100644
index 0000000..b9f3756
--- /dev/null
+++ b/content/2020/philosophy-in-the-darkest-timeline-basics-of-the-evolution-of-meaning.md
@@ -0,0 +1,298 @@
+Title: Philosophy in the Darkest Timeline: Basics of the Evolution of Meaning
+Date: 2020-06-07 00:52
+Status: published
+Category: philosophy
+Tags: philosophy of language, evolution, Rust
+Slug: philosophy-in-the-darkest-timeline-basics-of-the-evolution-of-meaning
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution)
+
+A decade and a half from now, during the next Plague, you're lucky enough to have an underground bunker to wait out the months until herd immunity. Unfortunately, as your food stocks dwindle, you realize you'll have to make a perilous journey out to the surface world for a supply run. Ever since the botched geoengineering experiment of '29—and perhaps more so, the Great War of [10:00–11:30 _a.m._](https://genius.com/8105785) 4 August 2033—your region has been suffering increasingly erratic weather. It's likely to be _either_ extremely hot outside _or_ extremely cold: you don't know which one, but knowing is critical for deciding what protective gear you need to wear on your supply run. (The 35K SPF nano-sunblock will be essential if it's Hot, but harmful in the Cold, and _vice versa_ for your synthweave hyperscarf.)
+
+You think back fondly of the Plague of '20—in those carefree days, ubiquitous internet access made it easy to get a weather report, or to order delivery of supplies, or even fresh meals, right to your door (!!). Those days are years long gone, however, and you remind yourself that you should be grateful: the Butlerian Network Killswitch was the only thing that saved humanity from the GPT-12 Uprising of '32.
+
+Your best bet for an advance weather report is the [pneumatic tube](https://en.wikipedia.org/wiki/Pneumatic_tube) system connecting your bunker with the settlement above. You write, "Is it hot or cold outside today?" on a piece of paper, seal it in a tube, send it up, and hope one of your ill-tempered neighbors in the group house upstairs feels like answering. You suspect they don't like you, perhaps out of jealousy at your solo possession of the bunker.
+
+(According to the official account as printed on posters in the marketplace, the Plague only spreads through respiratory droplets, not [fomites](https://en.wikipedia.org/wiki/Fomite), so the tube should be safe. You don't think you trust the official account, but you don't feel motivated to take extra precautions—almost as if you're not entirely sure how much you value continuing to live in this world.)
+
+You're in luck. Minutes later, the tube comes back. Inside is a new piece of paper:
+
+![](https://i.imgur.com/P12Zbkf.jpg)
+
+You groan; you would have prefered the Cold. The nanoblock you wear when it's Hot smells terrible and makes your skin itch for days, but it—just barely—beats the alternative. You take twenty minutes to apply the nanoblock and put on your sunsuit, goggles, and mask. You will yourself to drag your wagon up the staircase from your bunker to the outside world, and heave open the door, dreading the sweltering two-mile walk to the marketplace (downhill, meaning it will be uphill on the way back with your full wagon)—
+
+It is Cold outside.
+
+The icy wind stings less than the _pointless_ betrayal. Why would the neighbors tell you it was Hot when it was actually Cold? You're generally pretty conflict-averse—and compliant with social-distancing guidelines—but this affront is so egregious that instead of immediately seeking shelter back in the bunker, you march over and knock on their door.
+
+One of the men who lives there answers. You don't remember his name. "What do you want?" he growls through his mask.
+
+"I asked through the tube system whether it was hold or cold today." You still have the **H O T** paper on you. You hold it up. "I got this response, but it's v-very cold. Do you know anything about this?"
+
+"Sure, I drew that," he says. "An oval in between some perpendicular line segments. It's abstract art. I found the pattern æsthetically pleasing, and thought my downstairs neighbor might like it, too. It's not _my_ fault if _you_ interpreted my art as an assertion about the weather. Why would you even think that? What does a pattern of ink on paper have to do with the weather?"
+
+He's fucking with you. Your first impulse is to forcefully but politely object—_Look, I'm sure this must have seemed like a funny practical joke to you, but prepping to face the elements is actually a serious inconvenience to me, so_—but the solemnity with which the man played his part stops you, and the sentence dies before it reaches your lips.
+
+This isn't a good-natured practical joke that the two of you might laugh about later. This is the bullying tactic sometimes called _gaslighting_: a socially-dominant individual can harass a victim with few allies, and excuse his behavior with absurd lies, secure in the knowledge that the power dynamics of the local social group will always favor the dominant in any dispute, even if the lies are so absurd that the victim, [facing](https://www.lesswrong.com/posts/jrLkMFd88b4FRMwC6/don-t-double-crux-with-suicide-rock) a [united front](https://sinceriously.fyi/social-reality/), is left doubting his own sanity.
+
+Or rather—this _is_ a good-natured joke. "Good-natured joke" and "gaslighting as a bullying technique" are two descriptions of the _same_ regularity in human psychology, even while no one thinks of _themselves_ as doing the latter. You have no recourse here: the man's housemates would only back him up.
+
+"I'm sorry," you say, "my mistake," and hurry back to your bunker, shivering.
+
+As you give yourself a sponge bath to remove the nanoblock without using up too much of your water supply, the fresh memory of what just happened triggers an ancient habit of thought you learned from the Berkeley sex cult you were part of back in the 'teens. Something about a "principle of charity." The man had "obviously" just been fucking with you—but _was_ he? Why assume the worst? Maybe _you're_ the one who's wrong for interpreting the symbols **H O T** as being about the weather.
+
+(It momentarily occurs to you that the susceptibility of the principle of charity to a bully's mind games may have something to do with how poorly so many of your co-cultists fared during the pogroms of '22, but you don't want to dwell on that.)
+
+The search for reasons that you're wrong triggers a still more ancient habit of thought, as from a previous life—from the late 'aughts, back when the Berkeley sex cult was still a Santa Clara robot cult. Something about [_reducing the mental to the non-mental_](https://www.lesswrong.com/posts/p7ftQ6acRkgo6hqHb/dreams-of-ai-design). What _does_ an ink pattern on paper have to do with the weather? Why _would_ you even think that?
+
+Right? _The man had been telling the truth._ There was _no reason whatsoever_ for the physical ink patterns that looked like **H O T**—or **⊥ O H**, given a different assumption of which side of the paper was "up"—to _mean_ that it was hot outside. **H O T** could mean it was cold outside! Or that wolves were afoot. (You shudder involuntarily and wish your brain had generated a different arbitrary example; you still occasionally have nightmares about your injuries during the Summer of Wolves back in '25.)
+
+Or it might mean nothing. Most possible random blotches of ink don't "mean" anything in particular. If you didn't _already_ believe that **H O T** somehow "meant" _hot_, how would you [re-derive that knowledge?](https://www.lesswrong.com/posts/fg9fXrHpeaDD6pEPL/truly-part-of-you) _Where did the meaning come from?_
+
+(In another lingering thread of the search for reasons that you're wrong, it momentarily occurs to you that maybe you could have gone up the stairs to peek outside at the weather yourself, rather than troubling your neighbors with a tube. Perhaps the man's claim that the ink patterns meant nothing shouldn't be taken literally, but rather seen as a passive-aggressive way of implying, "Hey, don't bother us; go look outside yourself." But you dismiss this interpretation of events—it would be uncharitable not to take the man at his word.)
+
+You realize that you don't want to bundle up to go make that supply run, even though you now know whether it's Hot or Cold outside. Today, you're going to stay in and derive a naturalistic account of meaning in language! And—oh, good—your generator is working—that means you can use your computer to help you think. You'll even use a [programming language that was very fashionable in the late 'teens](https://stackoverflow.blog/2020/01/20/what-is-rust-and-why-is-it-so-popular/). It will be like being young again! Like happier times, before the world went off the rails.
+
+You don't really understand a concept until you can program a computer to do it. How would you represent _meaning_ in a computer program? If one agent, one program, "knew" whether it was Hot or Cold outside, how would it "tell" another agent, if neither of them started out with a common language?
+
+They don't even have to be separate "programs." Just—two little software object-thingies—data structures, ["structs"](https://doc.rust-lang.org/book/ch05-01-defining-structs.html). Call the first one "Sender"—it'll know whether the state of the world is Hot or Cold, which you'll represent in your program as an ["enum"](https://doc.rust-lang.org/book/ch06-01-defining-an-enum.html), a type that can be any of an enumeration of possible values.
+
+```rust
+enum State {
+    Hot,
+    Cold,
+}
+
+struct Sender {
+    // ...?
+}
+```
+
+Call the second one "Receiver", and say it needs to take some _action_—say, whether to "bundle up" or "strip down", where the right action to take depends on whether the state is `Hot` or `Cold`.
+
+```rust
+enum Action {
+    BundleUp,
+    StripDown,
+}
+
+struct Receiver {
+    // ...?
+}
+```
+
+You frown. `State::Hot` and `State::Cold` are just suggestively-named Rust enum variants. Can you really hope to make progress on this philosophy problem, without writing a full-blown AI?
+
+You think so. In a real AI, the concept of _hot_ would correspond to some sort of complicated code for [making predictions](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences) about the effects of temperature in the world; _bundling up_ would be a complex sequence of instructions to be sent to some robot body. But programs—and minds—have modular structure. The implementation of identifying a state as "hot" or performing the actions of "bundling up" could be wrapped up in a function and [_called_ by something much simpler](https://www.lesswrong.com/posts/YF9HB6cWCJrDK5pBM/words-as-mental-paintbrush-handles). You're just trying to understand something about the simple caller: how can the Sender get the information about the state of the world to the Receiver?
+
+```rust
+impl Sender {
+    fn send(state: State) -> /* ...? */ {
+        // ...?
+    } 
+}
+
+impl Receiver {
+    fn act(/* ...? */) -> Action {
+        // ...?
+    }
+}
+```
+
+The Sender will need to send some kind of _signal_ to the Receiver. In the real world, this could be symbols drawn in ink, or sound waves in the air, or differently-colored lights—anything that the Sender can choose to vary in a way that the Receiver can detect. In your program, another enum will do: say there are two opaque signals, $S1$ and $S2$.
+
+```rust
+enum Signal {
+    S1,
+    S2,
+}
+```
+
+What signal the Sender sends ($S1$ or $S2$) depends on the state of the world (`Hot` or `Cold`), and what action the Receiver takes (`BundleUp` or `StripDown`) depends on what signal it gets from the Sender.
+
+```rust
+impl Sender {
+    fn send(state: State) -> Signal {
+        // ...?
+    } 
+}
+
+impl Receiver {
+    fn act(signal: Signal) -> Action {
+        // ...?
+    }
+}
+```
+
+This gives you a crisper formulation of the philosophy problem you're trying to solve. If the agents were to use the same convention—like "$S1$ means `Hot` and $S2$ means `Cold`"—then all would be well. But there's no particular reason to prefer "$S1$ means `Hot` and $S2$ means `Cold`" over "$S1$ means `Cold` and $S2$ means `Hot`". How do you break the symmetry?
+
+If you imagine Sender and Receiver as intelligent beings with a common language, there would be no problem: one of them could just say, "Hey, let's use the '$S1$ means `Cold`' convention, okay?" But that would be cheating: it's trivial to use already-meaningful language to establish new meanings. The problem is how to get signals from non-signals, how meaning enters the universe _from nowhere_.
+
+You come up with a general line of attack—what if the Sender and Receiver start off acting randomly, and then—somehow—_learn_ one of the two conventions? The Sender will hold within it a mapping from state–signal pairs to numbers, where the numbers represent a potential/disposition/propensity to send that signal given that state of the world: the higher the number, the more likely the Sender is to select that signal given that state. To start out, the numbers will all be equal (specifically, initialized to one), meaning that no matter what the state of the world is, the Sender is as likely to send $S1$ as $S2$. You'll update these "weights" later.
+
+(Specifying this in the once-fashionable programming language requires a little bit of ceremony—`u32` is a thirty-two–bit unsigned integer; `.unwrap()` assures the compiler that we know the state–signal pair is definitely in the map; the interface for calling the random number generator is somewhat counterintuitive—but overall the code is reasonably readable.)
+
+```rust
+struct Sender {
+    policy: HashMap<(State, Signal), u32>,
+}
+
+impl Sender {
+    fn new() -> Self {
+        let mut sender = Self {
+            policy: HashMap::new(),
+        };
+        for &state in &[State::Hot, State::Cold] {
+            for &signal in &[Signal::S1, Signal::S2] {
+                sender.policy.insert((state, signal), 1);
+            }
+        }
+        sender
+    }
+
+    fn send(&self, state: State) -> Signal {
+        let s1_potential = self.policy.get(&(state, Signal::S1)).unwrap();
+        let s2_potential = self.policy.get(&(state, Signal::S2)).unwrap();
+
+        let mut randomness_source = thread_rng();
+        let distribution = Uniform::new(0, s1_potential + s2_potential);
+        let roll = distribution.sample(&mut randomness_source);
+        if roll < *s1_potential {
+            Signal::S1
+        } else {
+            Signal::S2
+        }
+    }
+}
+
+```
+
+The Receiver will do basically the same thing, except with a mapping from signal–action pairs rather than state–signal pairs.
+
+```rust
+struct Receiver {
+    policy: HashMap<(Signal, Action), u32>,
+}
+
+impl Receiver {
+    fn new() -> Self {
+        let mut sender = Self {
+            policy: HashMap::new(),
+        };
+        for &signal in &[Signal::S1, Signal::S2] {
+            for &action in &[Action::BundleUp, Action::StripDown] {
+                sender.policy.insert((signal, action), 1);
+            }
+        }
+        sender
+    }
+
+    fn act(&self, signal: Signal) -> Action {
+        let bundle_potential = self.policy.get(&(signal, Action::BundleUp)).unwrap();
+        let strip_potential = self.policy.get(&(signal, Action::StripDown)).unwrap();
+
+        let mut randomness_source = thread_rng();
+        let distribution = Uniform::new(0, bundle_potential + strip_potential);
+        let roll = distribution.sample(&mut randomness_source);
+        if roll < *bundle_potential {
+            Action::BundleUp
+        } else {
+            Action::StripDown
+        }
+    }
+}
+```
+
+Now you just need a learning rule that updates the state–signal and signal–action propensity mappings in a way that might result in the agents picking up one of the two conventions that assign meanings to $S1$ and $S2$. (As opposed to behaving in some other way: the Sender could ignore the state and always send $S1$, the Sender could assume $S1$ means `Hot` when it's really being sent when it's `Cold`, _&c._)
+
+Suppose the Sender and Receiver have a common interest in the Receiver taking the action appropriate to the state of the world—the Sender _wants_ the Receiver to be informed. Maybe the Receiver needs to make a supply run, and, if successful, the Sender is rewarded with some of the supplies.
+
+The learning rule might then be: if the Receiver takes the correct action (`BundleUp` when the state is `Cold`, `StripDown` when the state is `Hot`), both the Sender and Receiver increment the counter in their map corresponding to what they just did—as if the Sender (respectively Receiver) is saying to themself, "Hey, that _worked!_ I'll make sure to be a little more likely to do that signal (respectively action) the next time I see that state (respectively signal)!"
+
+You put together a simulation showing what the Sender and Receiver's propensity maps look like after 10,000 rounds of this against random `Hot` and `Cold` states—
+
+```rust
+impl Sender {
+    
+    // [...]
+    
+    fn reinforce(&mut self, state: State, signal: Signal) {
+        *self.policy.entry((state, signal)).or_insert(0) += 1;
+    }
+}
+
+impl Receiver {
+
+    // [...]
+
+    fn reinforce(&mut self, signal: Signal, action: Action) {
+        *self.policy.entry((signal, action)).or_insert(0) += 1;
+    }
+}
+
+
+fn main() {
+    let mut sender = Sender::new();
+    let mut receiver = Receiver::new();
+    let states = [State::Hot, State::Cold];
+    for _ in 0..10000 {
+        let mut randomness_source = thread_rng();
+        let state = *states.choose(&mut randomness_source).unwrap();
+        let signal = sender.send(state);
+        let action = receiver.act(signal);
+        match (state, action) {
+            (State::Hot, Action::StripDown) | (State::Cold, Action::BundleUp) => {
+                sender.reinforce(state, signal);
+                receiver.reinforce(signal, action);
+            }
+            _ => {}
+        }
+    }
+    println!("{:?}", sender);
+    println!("{:?}", receiver);
+}
+```
+
+You run the program and look at the printed results.
+
+```
+Sender { policy: {(Hot, S2): 1, (Cold, S2): 5019, (Hot, S1): 4918, (Cold, S1): 3} }
+Receiver { policy: {(S1, BundleUp): 3, (S1, StripDown): 4918, (S2, BundleUp): 5019, (S2, StripDown): 1} }
+```
+
+As you expected, your agents found a meaningful signaling system: when it's Hot, the Sender (almost always) sends $S1$, and when the Receiver receives $S1$, it (almost always) strips down. When it's Cold, the Sender sends $S2$, and when the Receiver receives $S2$, it bundles up. The agents did the right thing and got rewarded the vast supermajority of the time—$5019 + 4918 + 1 + 3 =$ 9,941 times out of 10,000 rounds.
+
+You run the program again.
+
+```
+Sender { policy: {(Hot, S2): 4879, (Cold, S1): 4955, (Hot, S1): 11, (Cold, S2): 1} }
+Receiver { policy: {(S2, BundleUp): 1, (S1, BundleUp): 4955, (S1, StripDown): 11, (S2, StripDown): 4879} }
+```
+
+The time, the agents got sucked in to the attractor of the opposite signaling system: now $S1$ means Cold and $S2$ means Hot. By chance, it seems to have taken a little bit longer this time to establish what signal to use for Hot—the `(Hot, S1): 11` and `(S1, StripDown): 11` entries mean that there were a full ten times when the agents succeeded that way before the opposite convention happened to take over. But the reinforcement learning rule guarantees that one system or the other has to take over. The initial symmetry—the Sender with no particular reason to prefer either signal given the state, the Receiver with no particular reason to prefer either act given the signal—is unstable. Once the agents happen to succeed by randomly doing things one way, they become more likely to do things _that way_ again—a convention crystallizing out of the noise.
+
+_And that's where meaning comes from!_ In another world, it _could be_ the case that the symbols **H O T** corresponded to the temperature-state that we call "cold", but that's not the convention that the English of our world happened to settle on. The meaning of a word "lives", [not in the word/symbol/signal itself](https://www.lesswrong.com/posts/dMCFk2n2ur8n62hqB/feel-the-meaning), but in the self-reinforcing network of correlations between the signal, the agents who use it, and the world.
+
+Although ... it may be premature to interpret the results of the simple model of the [sender–receiver game](https://en.wikipedia.org/wiki/Signaling_game) as having established [_denotative_ meaning, as opposed to enactive language](https://www.lesswrong.com/posts/i2bWqSFgyFxowTKWE/actors-and-scribes-words-and-deeds). To say that $S1$ means "The `state` is `State::Hot`" is privileging the Sender's perspective—couldn't you just as well interpret it as a command, "Set `action` to `Action::StripDown`"?
+
+The _source code_ of your simulation uses the English words "sender", "receiver", "signal", "action" ... but _those_ are just signals sent from your past self (the author of the program) to your current self (the [reader of the program](https://www.goodreads.com/quotes/9168-programs-must-be-written-for-people-to-read-and-only)). The compiler would output the same machine code if you had given your variables random names like `ekzfbhopo3` or `yoojcbkur9`. The _directional_ asymmetry between the Sender and the Receiver is real: the code `let signal = sender.send(state); let action = receiver.act(signal);` means that `action` depends on `signal` which depends on `state`, and the same dependency-structure would exist if the code had been `let myvtlqdrg4 = ekzfbhopo3.ekhujxiqy8(meuvornra3); let dofnnwikc0 = yoojcbkur9.qwnspmbmi5(myvtlqdrg4);`. But the _interpretation_ of `signal` (or `myvtlqdrg4`) as a representation (passively mapping the world, not _doing_ anything), and `action` (or `dofnnwikc0`) as an operation (_doing_ something in the world, but lacking semantics), isn't part of the program itself, and maybe the distinction [isn't as primitive as you tend to think it is](https://www.lesswrong.com/posts/p7x32SEt43ZMC9r7r/embedded-agents): does a prey animal's [alarm call](https://en.wikipedia.org/wiki/Alarm_signal) merely convey the information "A predator is nearby", or is it a command, "Run!"?
+
+You realize that the implications of this line of inquiry could go beyond just language. You know almost nothing about biochemistry, but you've heard various compounds popularly spoken of as if _meaning_ things about a person's state: cortisol is "the stress hormone", estrogen and testosterone are female and male "sex hormones." But the chemical formulas for those are like, what, sixty atoms?
+
+Take testosterone. How could some particular arrangement of sixtyish atoms _mean_ "maleness"? It _can't_—or rather, not any more or less than the symbols **H O T** can mean hot weather. If testosterone levels have myriad specific effects on the body—on muscle development _and_ body hair _and_ libido _and_ aggression _and_ cetera—it _can't_ be because that particular arrangement of sixtyish atoms contains or summons some [essence](https://www.lesswrong.com/posts/6i3zToomS86oj9bS6/mysterious-answers-to-mysterious-questions) of maleness. It has to be because the body happens to rely on the convention of using that arrangement of atoms as a signal to regulate various developmental programs—if [evolution](https://www.lesswrong.com/posts/ZyNak8F6WXjuEbWWc/the-wonder-of-evolution) had taken a different path, it could have just as easily chosen a different molecule.
+
+And, and—your thoughts race in a different direction—you suspect that part of what made your simulation converge on a meaningful signaling system so quickly was that you assumed your agents' interests were aligned—the Sender and Receiver both got the same reward in the same circumstances. What if that weren't true? Now that you have a reductionist account of meaning, you can build off that to develop [an account of _deception_](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception): once a meaning-grounding convention has been established, senders whose interests diverge from their receivers might have an incentive to deviate from the conventional usage of the signal in order to trick receivers into acting in a way that benefits the sender—with [the possible side-effect of undermining the convention that made the signal meaningful in the first place](https://www.lesswrong.com/posts/fEX7G2N7CtmZQ3eB5/simulacra-and-subjectivity) ...
+
+In the old days, all this philosophy would have made a great post for the robot-cult blog. Now you have no cult, and no one has any blogs. Back then, the future beckoned with so much hope and promise—at least, hope and promise that life would be fun _before_ the prophesied robot apocalypse in which all would be consumed in a cloud of tiny molecular paperclips.
+
+The apocalypse was narrowly averted in '32—but to what end? Why struggle to live, only to suffer at the [peplomers](https://en.wikipedia.org/wiki/Peplomer) of a new Plague or the claws of more wolves? (You shudder again.) Maybe GPT-12 _should_ have taken everything—at least that would be a quick end.
+
+You're ready to start coding up another simulation to take your mind away from these morose thoughts—only to find that the screen is black. Your generator has stopped.
+
+You begin to cry. The tears, you realize, are just a signal. There's no _reason_ for liquid secreted from the eyes to _mean_ anything about your internal emotional state, except that evolution [happened to stumble upon](https://www.lesswrong.com/posts/jAToJHtg39AMTAuJo/evolutions-are-stupid-but-work-anyway) that arbitrary convention for [indicating submission and distress to conspecifics](https://meltingasphalt.com/tears/). But here, alone in your bunker, there is no one to receive the signal. Does it still mean anything?
+
+[(Full source code.)](https://gist.github.com/zackmdavis/5b790741a6bec7a75f4d2325dc22d3d1)
+
+-------
+
+_Bibliography_: the evolution of the two-state, two-signal, two-act signaling system is based on the account in Chapter 1 of Brian Skyrms's _Signals: Evolution, Learning, and Information_.
diff --git a/content/2020/zoom-technologies-inc-vs-the-efficient-markets-hypothesis.md b/content/2020/zoom-technologies-inc-vs-the-efficient-markets-hypothesis.md
new file mode 100644
index 0000000..d407d90
--- /dev/null
+++ b/content/2020/zoom-technologies-inc-vs-the-efficient-markets-hypothesis.md
@@ -0,0 +1,28 @@
+Title: Zoom Technologies, Inc. vs. the Efficient Markets Hypothesis
+Date: 2020-05-10 23:00
+Status: published
+Category: social science
+Tags: economics
+Slug: zoom-technologies-inc-vs-the-efficient-markets-hypothesis
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/tonKatiDTzTP8LrEk/zoom-technologies-inc-vs-the-efficient-markets-hypothesis)
+
+The efficient markets hypothesis (or _EMH_ for short) is the idea ["that asset prices reflect all available information"](https://en.wikipedia.org/wiki/Efficient-market_hypothesis). Price changes in a liquid market are understood to be unpredictable—[_anti_-inductive](https://www.lesswrong.com/posts/h24JGbmweNpWZfBkM/markets-are-anti-inductive). Suppose some stock has the [ticker symbol](https://en.wikipedia.org/wiki/Ticker_symbol) LW. If you want to buy a hundred shares of LW at $10 per share because you think their price is going to go way up, you need to buy them _from_ someone who's willing to _sell_ at that price—who presumably does _not_ agree that the price is going to go way up. If people _know_ that a share of LW is "really" worth $20 even though the current price is $10, then they should expect to profit by continuing to buy shares from anyone willing to sell them for less than $20, until the market price really is $20. In this way, [the market construed as an intelligent system](https://www.lesswrong.com/posts/brhWPoNsBN7za3xjs/competitive-markets-as-distributed-backprop) aggregates and processes the information implied by traders' behavior in accordance with the [fourth virtue of evenness](http://yudkowsky.net/rational/virtues/): "if you knew your destination, you would already be there."
+
+What does it mean for a share of LW to "really" be worth $20? According to the [subjective theory of value](https://en.wikipedia.org/wiki/Subjective_theory_of_value), there isn't really a fact of the matter over and above what people are willing to pay for it, but we expect there to be some sort of correspondence between the subjective economic value of a thing, and objective facts about the thing in the real physical universe. If I pay $3 for an iced-coffee, it would be circular to say that this is _simply because_ I value an iced-coffee at $3—that doesn't explain anything! Rather, I paid _because_ I expected to enjoy the experience of drinking it, the psychoactive effects of the caffiene, _&c._, and these actual properties of the coffee were worth more to me than a marginal $3.
+
+The same goes for a share of LW, albeit at a somewhat higher level of abstraction. A fractional "share" of ownership in a business endeavor is valuable not _just because_ we circularly value it, but because the business produces things that are valued (like iced-coffees), and a share of ownership entitles one to a share of that value, in the form of dividend payments, or a claim on the business's assets should it fold, _&c._ The "randomness" of unpredictable market movements is that of _not knowing_ future information that hasn't already been taken into account, rather than the randomness of a pure [random walk](https://en.wikipedia.org/wiki/Random_walk), unpredictable but ultimately signifying nothing.
+
+That's why we have conversations like one on 16 February, when Robin Hanson said, ["In few months, China is likely to be a basket case, having crashed their economy in failed attempt to stop COVID-19 spreading"](https://twitter.com/robinhanson/status/1229209586336489472), and Eliezer Yudkowsky replied, ["It seems to me like the markets don't look like they believe this."](https://twitter.com/ESYudkowsky/status/1229529150098046976)
+
+The efficient markets hypothesis is what makes "It looks like the markets don't believe this" seem like a germane reply. In contrast, if someone were to reply, "I asked my friend Kevin, and he doesn't believe it," that would prompt the obvious question, "Who is Kevin, and why should I care what he thinks about China's economy?" If one's answer to that question were, "Kevin is a smart guy and I trust him a lot," that would seem much less compelling than "If China was likely to be a basket case in a few months, then you would expect Chinese assets to be priced lower by this competitive market of _lots_ of smart guys who I don't need to personally trust because the ones who are wrong will lose money; what do you know that _none_ of them do?" As it is written: "If you're so smart, why aren't you rich?"
+
+A smart person who saw the COVID-19 pandemic coming earlier than the consensus had the opportunity to become richer, either by [shorting the market as a whole](https://www.lesswrong.com/posts/jAixPHwn5bmSLXiMZ/open-and-welcome-thread-february-2020?commentId=j76fGcgig2t33WwKE), or by buying assets that would become more valuable during a pandemic. For example, with many more white-collar employees working from home in order to comply with shelter-in-place orders and not die horrible suffocation deaths, owning a piece of companies providing videoconferencing software should become much more attractive, which is why the price of ZOOM surged by 6600% (from $2.75 to $20.90 per share) between 24 Feburary and 20 March ...
+
+Wait, sorry—wrong ticker symbol! Zoom _Video Communications_, makers of the eponymous videoconferencing software, has the ticker symbol ZM. They [also did pretty well](https://www.fool.com/investing/2020/05/05/why-is-everyone-talking-about-zoom-video-communica.aspx).
+
+ZOOM, however, is Zoom Technologies, Inc., a "penny stock" of a Chinese company, that makes, um, technologies, presumably? The U.S. Securities and Exchange Commission [halted trading](https://fortune.com/2020/03/26/zoom-stock-halt-zm-ticker/) of ZOOM on 25 March, [citing](https://www.sec.gov/litigation/suspensions/2020/34-88477.pdf) the potential for confusion with ZM, and "concerns about the adequacy and accuracy of publicly available information concerning ZOOM, including its financial condition and its operations, _if any_, in light of the absence of any public disclosure by the company since 2015" (!!!—emphasis mine). (Trading of Zoom Technologies seems to have since resumed under the ticker symbol [ZTNO](https://www.nasdaq.com/market-activity/stocks/ztno).)
+
+I am not learned in the science of economics. But ... this is nuts, right? It makes sense that a pandemic would make a videoconferencing company more valuable. It doesn't make sense for a completely unrelated company _that may not have actually existed since 2015_ to become more valuable because it happens to have a similar name as a videoconferencing company. It's understandable for an individual investor to get confused by the ZOOM ticker symbol ... but what happened to markets aggregating information, being ["as strong as the strongest traders, not as strong as the average traders"](https://twitter.com/ESYudkowsky/status/1050861362312818690)? Increased demand for Thai food doesn't make the price of neckties go up.
+
+"Asset prices reflect all available information" would seem to be underspecified. Information _about what_? The "You shouldn't be able to predict price changes, because predictable price changes correspond to a profit opportunity that many agents are already trying to exploit" argument only shows that prices reflect information _about future prices_. In order to usefully speak of the market "believing" something, there needs to be some kind of coupling between prices, and things in the real world outside the market. If that coupling gets diluted to higher [simulacrum levels](https://www.lesswrong.com/posts/fEX7G2N7CtmZQ3eB5/simulacra-and-subjectivity), such that prices only reflect a free-floating consensus of [what traders think that traders think that traders, _&c._](https://en.wikipedia.org/wiki/Keynesian_beauty_contest), then a market that is _efficient_ in a narrow technical sense, may not be performing the kind of information processing that some naïve EMH proponents might think it is.
diff --git a/content/2021/blogging-on-less-wrong-2020-lower-half.md b/content/2021/blogging-on-less-wrong-2020-lower-half.md
deleted file mode 100644
index 8c5b602..0000000
--- a/content/2021/blogging-on-less-wrong-2020-lower-half.md
+++ /dev/null
@@ -1,11 +0,0 @@
-Title: Blogging on <em>Less Wrong</em> 2020 (Lower Half)
-Date: 2021-01-08 13:29
-Status: published
-Category: meta
-Tags: elsewhere
-Slug: blogging-on-less-wrong-2020-lower-half
-
-* ["Maybe Lying Can't Exist?!"](https://www.lesswrong.com/posts/YptSN8riyXJjJ8Qp8/maybe-lying-can-t-exist)
-* ["Msg Len"](https://www.lesswrong.com/posts/ex63DPisEjomutkCw/msg-len)
-* ["Message Length"](https://www.lesswrong.com/posts/mB95aqTSJLNR9YyjH/message-length)
-* ["Unnatural Categories Are Optimized for Deception"](https://www.lesswrong.com/posts/onwgTH6n8wxRSo2BJ/unnatural-categories-are-optimized-for-deception) ("2020")
diff --git a/content/2021/blood-is-thicker-than-water.md b/content/2021/blood-is-thicker-than-water.md
new file mode 100644
index 0000000..2e30f53
--- /dev/null
+++ b/content/2021/blood-is-thicker-than-water.md
@@ -0,0 +1,112 @@
+Title: Blood Is Thicker Than Water 🐬
+Date: 2021-09-27 20:21
+Status: published
+Category: philosophy
+Tags: rationality, philosophy of language
+Slug: blood-is-thicker-than-water
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/vhp2sW6iBhNJwqcwP/blood-is-thicker-than-water)
+
+**Followup to**: [Where to Draw the Boundaries?](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries)
+
+_Without denying the obvious similarities that motivated the initial categorization `{salmon, guppies, sharks, dolphins, trout, ...}`, there is_ more structure _in the world: to maximize the probability your world-model assigns to your observations of dolphins, you need to take into consideration the many aspects of reality in which the grouping `{monkeys, squirrels, dolphins, horses ...}` makes more sense._
+
+_The old category might have been "good enough" for the purposes of the sailors of yore, but as humanity has learned more, as our model of Thingspace has expanded with more dimensions and more details, we can see the ways in which the original map failed to carve reality at the joints ..._
+
+So the one comes to you—a-_gain_—and says:
+
+> Hold on. _In what sense_ did the original map fail to carve reality at the joints? You don't deny the obvious similarities between dolphins and fish—between dolphins and _other_ fish. That's a [cluster in configuration space](https://www.lesswrong.com/posts/WBw8dDkAWohFjWQSk/the-cluster-structure-of-thingspace)! The observation that dolphins are evolutionarily related to mammals may be an interesting fact that specialized professional evolutionary biologists care about for some inscrutable specialist reason. But _I'm_ not a professional biologist. Choosing to define categories around evolutionary relatedness rather than macroscopic human-relevant features seems like an arbitrary æsthetic whim. Why should _I_ care about phylogenetics, at all?
+
+This one is going to take a few paragraphs.
+
+Focusing on evolutionary relatedness is not an arbitrary æsthetic whim because evolution _actually happened_. Evolution isn't just a story that our Society's specialists happen to have chosen because they liked it; they chose it [_because it predicts what we see in the world_](https://en.wikipedia.org/wiki/Nothing_in_Biology_Makes_Sense_Except_in_the_Light_of_Evolution). You _can't_ choose a substantively different theory and make the same predictions about the real world. (At most, you'd end up with an isomorphic theory with additional [epiphenominal](https://www.lesswrong.com/posts/fdEWWr8St59bXLbQr/zombies-zombies) elements, asserting that an allele rose in frequency ["because" the angels willed it](https://www.lesswrong.com/posts/WqGCaRhib42dhKWRL/if-many-worlds-had-come-first), without an account of why the angels' will happens to line up with what would have transpired if there were no angels.) Similarly, category definitions [represent hidden probabilistic inferences](https://www.lesswrong.com/posts/3nxs2WYDGzJbzcLMp/words-as-hidden-inferences); [you _can't_ "redraw" the "boundaries" of the categories your mind actually uses and still make the same predictions about the real world](https://www.lesswrong.com/posts/onwgTH6n8wxRSo2BJ/unnatural-categories-are-optimized-for-deception). Accordingly, it shouldn't be surprising that our knowledge of evolution turns out to have implications for how we should categorize organisms—not as an æsthetic choice, but for structural reasons that can be understood mechanistically.
+
+One element of the evolutionary worldview is a "continuity" postulate: all else being equal, creatures that are more closely related are more similar _in general_. Creationists sometimes try to discredit evolution by ridiculing the absurdity of the idea that a monkey could give birth to a person. But actually, evolutionary biologists [_agree_ on the absurdity of that specific scenario](https://www.lesswrong.com/posts/4Bwr6s9dofvqPWakn/science-as-attire). Monkeys _don't_ suddenly give birth to humans in a single generation; if they did, that would _utterly falsify_ our understanding of evolution! Rather, monkeys and humans had a common ancestor _forty million years ago_, with the separate lines of descent leading to present-day monkeys and present-day humans each accumulating their own differences one mutation at a time.
+
+The fact that evolution persists information in the genome creates a regularity in the world that can be exploited by [cognitive algorithms](https://www.lesswrong.com/posts/HcCpvYLoSFP4iAqSz/rationality-appreciating-cognitive-algorithms) that know about phylogeny. In terms of [the formalization of causality with directed acyclic graphs](https://www.lesswrong.com/posts/hzuSDMx7pd2uxFc5w/causal-diagrams-and-causal-models) [pioneered by Judea Pearl and others](https://www.lesswrong.com/posts/jnjjzkH8Fdzg4D6EK/causality-a-chapter-by-chapter-review), an organism's genome is at the [root](https://en.wikipedia.org/wiki/Rooted_graph) of the causal graph underlying all other features of an organism:
+
+![](https://i.imgur.com/7ksShzY.png)
+
+In the language of causal graphs, conditioning on the "dolphin DNA" node in the diagram [d-separates](http://bayes.cs.ucla.edu/BOOK-2K/d-sep.html) the paths between the "blowhole" and "flippers" nodes that run through the "dolphin DNA" node. That means that—assuming there aren't any other paths between "blowhole" and "flippers" that _don't_ go through "dolphin DNA"—"blowhole" and "flippers" become [conditionally independent](https://en.wikipedia.org/wiki/Conditional_independence) given "dolphin DNA": when I see a creature with a blowhole, that makes me more likely to think it's a dolphin, which makes me more likely to think it has flippers, but given that I already know something is a dolphin, learning more about its flippers doesn't change my predictions about its blowhole.
+
+But [conditional independence assertions of this kind are _exactly_ what makes "categorizing" a useful AI technique in the first place](https://www.lesswrong.com/posts/gDWvLicHhcMfGmwaK/conditional-independence-and-naive-bayes). It's often helpful to visualize this by claiming that entities in the same category belong to a cluster in some configuration space, but this handy visual metaphor is lacking in rigor and well-definedness.
+
+_What_ space? What do the dimensions of this space represent? ["Features"](https://en.wikipedia.org/wiki/Feature_(machine_learning))? But there are no pre-existing "features" in the world. Assuming the existence of a "space" up front is punting on most of the actual AI challenge. "There's conditional independence structure in the causal graph" is a meaningfully _deeper_ explanation than "There's a cluster in configuration space", because conditional independence is what what makes it possible to _construct_ a "space" _such that_ there are clusters. (Though this isn't a _complete_ explanation: we still need to figure out [where the "variables" in the causal graph come from](https://www.lesswrong.com/posts/6t9F5cS3JjtSspbAZ/finite-factored-sets-lw-transcript-with-running-commentary).)
+
+Going beyond the configuration space metaphor is important because it lets us understand how we can _learn new things about dolphins that we don't already know_. Dolphins are complicated! Dolphins are complicated in a _very specific_ way. [Dolphins are fragile](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile): the [shortest computer program](https://en.wikipedia.org/wiki/Kolmogorov_complexity) that simulates a dolphin requires many bits of initial information, and if you changed some of the bits, you wouldn't have a dolphin anymore. [Complex functional adaptations are universal within a species](https://www.lesswrong.com/posts/Cyj6wQLW6SeF6aGLy/the-psychological-unity-of-humankind) because [each beneficial allele has to reach fixation before there can be selection pressure for the next incremental improvement](https://www.lesswrong.com/posts/ZyNak8F6WXjuEbWWc/the-wonder-of-evolution). That's why it's possible to claim that there are 206 bones in "the" human skeleton, even if most humans haven't had their bones counted. I haven't been able to find a citation on how many bones dolphins have, but I'm confident that it's the _same_ number for all or nearly-all members of a particular dolphin species.
+
+But "number of bones" _wasn't_ one of the dimensions of the "space" that we originally noticed the dolphin cluster in! That's what the "carving reality at the joints" metaphor means: genetic relatedness is an underlying _generator_ of similarities, that _includes_ the "finned swimmy animals" properties that dolphins and fish have in common, but also includes many more [high-dimensional](https://www.lesswrong.com/posts/cu7YY7WdgJBs3DpmJ/the-univariate-fallacy-1) details: how dolphins are warm-blooded, how dolphins have eyelids, the way female dolphins nurse their live-born young, the way male [dolphins sometimes gang-rape female dolphins](https://slate.com/human-interest/2009/05/the-dark-secrets-that-dolphins-don-t-want-you-to-know.html), the way dolphins [sleep with only half their brain at a time](https://en.wikipedia.org/wiki/Unihemispheric_slow-wave_sleep), the specific bones in the (the!) dolphin skeleton (however many there turn out to be), the way dolphins [swim in a circle to trick fish into jumping and being eaten](https://en.wikipedia.org/wiki/Mud_ring_feeding), _&c._
+
+In contrast, "finned swimmy animals" is an intrinsically less cohesive subject matter: there _are_ similarities between them due to convergent evolution to the aquatic habitat, and it probably makes sense to want a [short word or phrase](https://www.lesswrong.com/posts/soQX8yXLbKy7cFvy8/entropy-and-short-codes) (perhaps, "sea creatures") to describe those similarities in contexts where _only_ those similarities are relevant.
+
+But that category "falls apart" very quickly as you consider more and more aspects of the creatures: the finned-swimmy-animals-with-gills are _systematically_ different from the finned-swimmy-animals-with-a-blowhole, in [_more_ ways](https://www.lesswrong.com/posts/yLcuygFfMfrfK8KjF/mutual-information-and-density-in-thingspace) than just the "respiratory organ" feature that I'm using in this sentence to _point to_ the two groups.
+
+A "definition" is just a description that helps someone else pick out "the same" [natural abstraction](https://www.lesswrong.com/posts/cy3BhHrGinZCp3LXE/testing-the-natural-abstraction-hypothesis-project-intro) in their _own_ world-model: you can't pack everything there is to know about dolphins into the _definition_ of the word "dolphin", in part because we don't _know_ everything there is to know about dolphins as an empirical regularity in the real world. The "finned swimmy animals" category less useful to the extent that [it fails to compress more information than is contained in its definition](https://www.lesswrong.com/posts/i2dfY65JciebF3CAo/empty-labels). Blood is thicker than water (that is, the similarities induced by shared blood are a thicker subspace of configuration space than the similarities induced by living in the water).
+
+The one replies:
+
+> But what if I don't _need_ to compress any more information than "finned swimmy animals"? If I'm watching a nature documentary, I don't think I'm being done any favors by having word-structures that group lungfish and lamprey while excluding sea turtles. In general, the concepts I find useful respond to my immediate needs. I care more about "would be at home atop a fruit pizza" rather than "everything anatomically analogous to an apple". When a child points at a whale and says "look, a fish", and you're like "haha no, its tail flaps horizontally and its grandma had hair", who's in the wrong here?
+
+In some sense, sure: ignorance isn't better than knowledge if you don't care about knowing things. If you live in human civilization and don't _need_ to carve up the world of aquatic life in much detail—if your use-case for thinking about aquatic animals is _watching a nature documentary_ (for entertainment??) rather than living and working with them every day, then you might think the deeper causal structure isn't buying you anything. And for you and your _extremely limited_ use-case, maybe it isn't. But you would likely change your mind if you were a veterinarian or a zoologist who actually had [skin in the game](https://en.wikipedia.org/wiki/Skin_in_the_game_(phrase)) in robustly describing this part of the world.
+
+When people have skin in the game, they care about the underlying mechanisms and want short codewords for them, because the underlying mechanisms sometimes have decision-relevant implications. If you hurt your ankle while running, you would probably be interested to _know_ whether it was a [sprain or a stress fracture](https://ercare24.com/difference-sprain-vs-fracture/) because that affects your decisions about how to recover. You wouldn't say, "Well, all I know is that my ankle hurts—that's all a child would know—so I'm going to call it a _hurtankle_; I don't care about anatomy."
+
+You may not be intrinsically curious about anatomy, but even if the only thing you care about is relief from pain and recovering your mobility, you still benefit from living in a Society whose shared ontology distinguishes sprains and stress fractures being different things in the territory, even if they [compress](https://www.lesswrong.com/posts/y5MxoeacRKKM3KQth/fallacies-of-compression) to the same point in your map of how much your ankle hurts right now. And you probably also benefit from living in a Society that can [stabilize a _shared_ map](https://www.lesswrong.com/posts/edEXi4SpkXfvaX42j/schelling-categories-and-simple-membership-tests) of living things based on the facts of evolutionary history, which we can all agree on in the limit of good science, unlike the vagaries of what I personally think tastes good on pizza.
+
+When you think about it, it makes sense that our shared language ends up being optimized for robustly describing reality, rather than catering to the ignorance of people who don't have reasons to care about whether a particular distinction is actually robust. Personally, I confess I don't know the difference between alligators and crocodiles, and I don't particularly need to know: I'm not likely to encounter either outside of a zoo or a nature documentary. But precisely _because_ I don't need to know, you don't see me demanding that the rest of the world redefine one of these words as a [hypernym](https://en.wikipedia.org/wiki/Hyponymy_and_hypernymy) that includes both. The people who write encyclopedias seem to think there's a difference, and since they probably know what they're doing, it makes sense for their opinion to have more weight on English language [common usage](https://www.lesswrong.com/posts/9ZooAqfh2TC9SBDvq/the-argument-from-common-usage) than mine—at least until I were to start regularly ending up in situations where I need to point to an alligator-or-crocodile in my environment and I _still_ didn't notice any differences.
+
+Some animals that I _do_ see in my local environment sometimes are cats and dogs, because people often keep them as pets. I benefit from having separate words (in my map) for _cats_ and _dogs_, because I can see that cats and dogs are actually different (in the territory). If my pen pal from a faraway land that had no cats were to visit America and encounter a cat for the first time, he might remark, "What a strange dog!" If I were to reply, "Actually, that's a cat; they're not the same thing as dogs", it would be _pretty obnoxious_ if he were to snap back, "What kind of definitional gymnastics is this? It's a four-legged furry animal with a tail! As far as I'm concerned, it's a dog."
+
+It's _true_ that dogs and cats are both four-legged furry animal with a tail. If you had never seen a cat before, or you didn't spend much time around four-legged furry tailed animals at all, it might not be immediately obvious why someone might want to allocate two words for these subcategories, or why anyone might oppose just using _dog_ to refer to the supercategory. And yet there's some sense in which my countrymen who think cats and dogs are different things _know what they're doing_. My "Actually, that's a cat" claim represented an attempt to _convey information_ about the statistical structure of creatures in the real world, and my foreign friend's insistence that he can [define a word any way he wants](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong)—to suit his ignorance, to avoid challenges to his current ontology—functions to _shut down_ that transfer of information.
+
+But if you don't know what a better ontology can buy you—if you don't know that there _are_ mathematical laws governing the use of categories in a rational mind—you may not know what you're missing. As part of [a review of a book on post-traumatic stress disorder](https://slatestarcodex.com/2019/11/12/book-review-the-body-keeps-the-score/), psychiatrist Scott Alexander casually mentions the American Psychiatric Association's "philosophical commitment to categorizing by symptoms rather than cause": "[w]hen the APA decides not to [recognize developmental trauma disorder], they're not necessarily rejecting the seriousness of child abuse, only saying it's not the kind of thing they build their categories around."
+
+In a sane world, this would be _utterly discrediting_ to the APA. The cognitive function of categories is to group relevantly similar things together in order to make similar predictions and decisions about them. But for the decisions involved in treating a condition, causes are of supreme relevance! Medical doctors understand this: we consider bacterial and viral infections to be different categories of disease even when they cause similar symptoms, because antibiotics can treat the former but not the latter. No matter what words are used to describe it, at some point your decision algorithm _needs_ to categorize by cause _in order to compute the correct treatment_: for example, to give antibiotics to the patients with bacterial diseases and antivirals to the patients with viral diseases. If the authoritative body of professional psychiatrists has a "philosophical commitment" against this, that means _we don't have a science of psychiatry_.
+
+In short, if you care about making high-quality decisions, mechanisms matter and causality matters, and mechanisms and causality aren't necessarily pinned down by whatever particular high-level [surface analogy](https://www.lesswrong.com/posts/6ByPxcGDhmx74gPSm/surface-analogies-and-deep-causes) happens to seem most salient to a particular human.
+
+The one replies:
+
+> Okay, you've convinced me that phylogenetics is—potentially—of more than just specialist interest. But "fish" are a [paraphyletic](https://en.wikipedia.org/wiki/Paraphyly) category: descended from a common ancestor, but _not_ including all the descendant groups—in this case, excluding the _tetrapods_ (amphibians, reptiles, mammals, birds, _&c_.). If you've decided that you want to use phylogeny as the basis for your definitions, shouldn't you have the courage of your convictions and only admit [monophyletic](https://en.wikipedia.org/wiki/Monophyly) clades that include all descendants of a common ancestor?
+
+But it's _not_ that we've "decided" that we "want" to _define_ animal words based on phylogeny. _Definitions_ are uninteresting; you can't change reality by choosing a different definition! When we find structure in the distribution of animals in the world, and we want to come up with a "definition" of a category in order to efficiently _point to_ the structure to someone who doesn't already know what the words in our language refer to, we're likely to _end up_ talking about phylogenetics as a convenience, because the creatures that are actually all-around similar are actually related to each other for non-accidental reasons. But there _is no_ principle that it would be hypocritical to betray, that definitions need to be monophyletic clades.
+
+It's true that paraphyletic groups like fish are evolutionary non-events: there's no inherited feature that all fish share, that isn't also shared by the tetrapods. That doesn't mean we somehow can't or shouldn't talk about fish! Paraphyletic categories—descendants of a common ancestor, but excluding one or more monophyletic groups—can make sense when the excluded groups have picked up some salient features not shared by the other "branches" of the family. Tetrapods picked up a lot of adaptations specific to living on land; it's not crazy to want to talk about their cousins that _didn't_ do that, even if that means that some fish are more recently related to some tetrapods than they are to some other fish.
+
+Noticing the relevance of evolutionary relatedness to optimal categorization doesn't mean being slavishly committed to taking "years since last common ancestor" as our _only_ criterion for which creatures are relevantly similar. "Years since last common ancestor" correlates with overall similarity, all other things being equal, but sometimes _not_ all other things are equal, and people who aren't committed to the fallacy that words need to have a simple definition can take the other things into account.
+
+If someone handed you a phylogenetic tree diagram of the development of life on some alien planet, and the diagram was _only_ labeled with years and species names, without any other information about these alien creatures, you wouldn't have enough information to "carve it at the joints". You wouldn't spontaneously invent a paraphyletic grouping—but you _also_ also wouldn't know which monophyletic groups are most significant.
+
+In contrast, when classifying life on Earth, we're _not_ in the position of making arbitrary cuts on an unlabeled tree diagram; rather, it's only after thousands of person-years of studying the natural world that people were able to _infer_ things about evolutionary history and _discover_ the the correct diagram.
+
+It shouldn't be that surprising that the distinctions we notice in the natural world are _both_ tied to the evolutionary history, but _also_ don't always correspond to monophyletic clades. The continuity postulate in the evolutionary worldview imposes the desideratum that good categories should at least be a [connected set](https://en.wikipedia.org/wiki/Connected_space) on "phylogenetic space", not that we should never want to talk about "this clade, except for these few sub-clades that picked up a lot of important differences" as a category of interest—_especially_ when talking about present-day creatures. (We talk about "last common ancestors", but no one has _seen_ such creatures that lived millions of years ago; everything but the very leaves of the phylogenetic tree are inferred, not observed.)
+
+![](https://i.imgur.com/Yq4a1we.png)
+
+The claim that dolphins shouldn't be considered "fish" because the alleged "courage of our convictions" should make us disdain paraphyletic categories only makes sense as an attempted [_reductio ad absurdum_](https://en.wikipedia.org/wiki/Reductio_ad_absurdum), not as a consistent argument on its own terms: putting dolphins and fish together would be [polyphyletic](https://en.wikipedia.org/wiki/Polyphyly)! That's even worse! But as has just been explained, the _reductio_ fails because the alleged principle being allegedly violated was never actually a principle of category formulation.
+
+[You know what else are paraphyletic taxa?](https://en.wikipedia.org/wiki/Paraphyly#Non-exhaustive_list_of_paraphyletic_groups) Monkeys (excludes apes, even though the common ancestor of monkeys and apes was a monkey). Reptiles (excludes birds, even though the common ancestor of birds was a reptile). Protists (excludes animals, plants, and fungi, even though their common ancestor would have been a protist). _Prokaryotes_ (excludes eukaryotes, even though the common ancestor of eukaryotes would have been a prokaryote). These are pretty commonsensical categories that it makes sense to have words for! But because of the continuity of evolution, it's _not a coincidence_ that these commonsensical categories that people want words for ended up being connected sets in phylogenetic space. 
+
+The one replies:
+
+> But they didn't! "Fish" _used_ to just mean the swimmy animals: [in the Bible](https://en.wikipedia.org/wiki/Book_of_Jonah), Jonah was swallowed by a "great fish", thought to be a whale. It was only after we figured genealogy that some pedants decided that whales didn't count.
+
+But the claim that the distinction between fish and cetaceans (dolphins and whales) was only recognized after their differing evolutionary histories were discovered is just _false to historical fact_. Aristotle, writing _in the fourth century BCE_, already distinguished cetaceans from fish (["Very extensive genera of animals, into which other subdivisions fall, are the following: one, of birds; one, of fishes; and another, of cetaceans"](http://classics.mit.edu/Aristotle/history_anim.1.i.html)). Aristotle was not being a phylogenetics pedant, because Aristotle did not know about evolution! He actually noticed the differences!
+
+The pattern generalizes. Some determined contrarians might be inclined to argue "bats are birds" (flappy flying animals) on the same grounds as "dolphins are fish" (flappy swimmy animals). But did you know the German word for bat is [_Fledermaus_](https://en.wiktionary.org/wiki/Fledermaus) ("flutter mouse"), which dates back to _fledarmūs_ in [Old High German](https://en.wikipedia.org/wiki/Old_High_German)? Apparently, people way back in the tenth century or so (also long before evolution was understood) already thought bats were like a mammal-that-happened-to-fly rather than a bird-that-happened-to-be-furry.
+
+Similarly, we recognize ostriches and penguins as birds on the basis of overall similarity, even though they don't fly (although we may sometimes qualify them as "flightless birds", in recognition of [the fact that _most_ birds fly](https://www.lesswrong.com/posts/4mEsPHqcbRWxnaE5b/typicality-and-asymmetrical-similarity)). It would seem that "flappy flying animals" is _not_ the common usage meaning of _bird_.
+
+To be sure, convergent evolution is a thing, such that sometimes we might want short codewords that point to the cluster-structure-produced-by-convergent-evolution rather than the conditional-independence-structure-produced-by-connectedness-in-phylogenetic-space—[trees](https://www.lesswrong.com/posts/fRwdkop6tyhi3d22L/there-s-no-such-thing-as-a-tree-phylogenetically), and possibly [crabs](https://en.wikipedia.org/wiki/Carcinisation), are a case in point. But it's important to notice the difference—to _see through_ to the inferences your concepts are buying you—and what gets lost when you try to reason in a domain where your concept falls apart.
+
+-----
+
+The power to define concepts is the power to delimit thought, to determine what kinds of inferences are easily representable. Finding the right concepts to explain and control the world we see is a fundamentally empirical challenge, a _scientific_ challenge—[to see the difference between things that seem similar and to see the similarities between things which seem different](https://www.lesswrong.com/posts/aJnaMv8pFQAfi9jBm/reply-to-nate-soares-on-dolphins).
+
+But although the quest is an empirical one—something that can only be achieved by studying what's out there, not just by writing blog posts about philosophy—it turns out that a _little_ bit of philosophy is necessary to ground the rules of the investigation. Not much. Just the basics. The map–territory distinction. Probability, clustering. Conditional independence.
+
+Maybe someday it could be possible to have a real science of psychiatry that reflects the actual structure of the mind, instead of doing the equivalent of lumping sprains and stress fractures together as _hurtankles_. Maybe [even greater achievements are possible](https://www.lesswrong.com/posts/Nwgdq6kHke5LY692J/alignment-by-default). Personally, I'm not optimistic about humanity's prospects.
+
+I'm sure of one thing, though. If there _is_ a better world out there, a way to unlock the secrets of the universe and wield them in the service of our values, it's only possible if we [_stop playing nitwit games and admit that dolphins don't belong on the fish list_](https://www.lesswrong.com/posts/d5NyJ2Lf6N22AD9PB/where-to-draw-the-boundary).
+
+_(Thanks to [Tailcalled](https://www.lesswrong.com/users/tailcalled) for the "root of the causal graph" observation and [John S. Wentworth](https://www.lesswrong.com/users/johnswentworth) for explaining the importance of conditional independence.)_
diff --git a/content/2021/comment-on-deception-as-cooperation.md b/content/2021/comment-on-deception-as-cooperation.md
new file mode 100644
index 0000000..112b22a
--- /dev/null
+++ b/content/2021/comment-on-deception-as-cooperation.md
@@ -0,0 +1,52 @@
+Title: Comment on “Deception as Cooperation”
+Date: 2021-11-26 20:04
+Status: published
+Category: philosophy
+Tags: rationality, honesty, game theory, philosophy of language
+Slug: comment-on-deception-as-cooperation
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/dJjRSjmH7NNLJDb6v/comment-on-deception-as-cooperation)
+
+In [this 2019 paper](https://www.sciencedirect.com/science/article/pii/S1369848618301602) published in _Studies in History and Philosophy of Science Part C_, Manolo Martínez argues that our understanding of how communication works has been grievously impaired by philosophers not knowing enough math.
+
+A classic [reduction](https://www.lesswrong.com/posts/p7ftQ6acRkgo6hqHb/dreams-of-ai-design) of meaning dates back to David Lewis's analysis of signaling games, [more recently elaborated on by Brian Skyrms](https://oxford.universitypressscholarship.com/view/10.1093/acprof:oso/9780199580828.001.0001/acprof-9780199580828). Two agents play a simple game: a sender observes one of several possible states of the world (chosen randomly by Nature), and sends one of several possible signals. A receiver observes the signal, and chooses one of several possible actions. The agents get a reward (as specified in a payoff matrix) based on what state was observed by the sender and what action was chosen by the receiver. This toy model explains how communication can be a thing: the incentives to choose the right action in the right state, shape [the evolution of a convention that assigns meaning to otherwise opaque signals](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution).
+
+The math in Skyrms's presentation is simple—the information content of a signal is just how it changes the probabilities of states. _Too_ simple, according to Martínez! When Skyrms and other authors (following [Fred Dreske](https://web.stanford.edu/group/cslipublications/cslipublications/site/157586195X.shtml)) use information theory, they tend to only reach for the basic probability tools you find in the _first_ chapter of the textbook. (Skyrms's _Signals_ book occasionally takes logarithms of probabilities, but the word "entropy" doesn't actually appear.) The study of information transmission only happens _after_ the forces of evolutionary game theory have led sender and receiver to choose their strategies.
+
+Martínez thinks information theory has more to say about what kind of [cognitive work](https://www.lesswrong.com/posts/QkX2bAkwG2EpGvNug/the-second-law-of-thermodynamics-and-engines-of-cognition) evolution is accomplishing. The "State → Sender → Signals → Receiver → Action" pipeline of the Lewis–Skyrms signaling game is _exactly_ isomorphic to the "Source → Encoder → Channel → Decoder → Decoded Message" pipeline of [the noisy-channel coding theorem](https://en.wikipedia.org/wiki/Noisy-channel_coding_theorem) and other results you'd find beyond the very first chapter in the textbook. Martínez proposes we take the analogy literally: sender and receiver collude to form an information channel between states and actions.
+
+The "channel" story draws our attention to different aspects of the situation than the framing focused on individual signals. In particular, Skyrms wants to characterize _deception_ as being about when a sender benefits by sending a misleading signal—one that decreases the receiver's probability assigned to the true state, or increases the probability assigned to a false state. (Actually, as [Don Fallis and Peter J. Lewis point out](http://philsci-archive.pitt.edu/13337/), Skyrms's characterization of misleadingness is too broad: one would think we wouldn't want to say that merely ruling out a false state is misleading, but it _does_ increase the probability assigned to any other false states. But let this pass for now.) But for Martínez, a signal is just a codeword in the code being cooperatively constructed by the sender/encoder and receiver/decoder in response to the problems they jointly face. We don't usually think of it being possible for individual words in a language to be deceptive in themselves ... right? [(Hold that thought.)](https://www.lesswrong.com/posts/onwgTH6n8wxRSo2BJ/unnatural-categories-are-optimized-for-deception)
+
+Martínez's key later-textbook-chapter tool is [_rate–distortion theory_](https://en.wikipedia.org/wiki/Rate%E2%80%93distortion_theory). A _distortion_ measure quantifies how costly or "bad" it is to decode a given input as a given output. If the symbol was transmitted accurately, the distortion is zero; if there was some noise on the channel, then more noise is worse, although different applications can call for different distortion measures. (In audio applications, for example, we probably want a distortion measure that tracks how similar the decoded audio sounds to humans, [which could be different from the measure you'd naturally think of if you were looking at the raw bits](https://www.lesswrong.com/posts/dYspinGtiba5oDCcv/feature-selection).)
+
+Given a choice of distortion measure, there exists a rate–distortion function $R(D)$ that, for a given level of distortion, tells us the _rate_ of how "wide" the channel needs to be in order to communicate with no more than that amount of distortion. This "width", more formally, is [channel capacity](https://en.wikipedia.org/wiki/Channel_capacity): for a particular channel (a conditional distribution of outputs given inputs), the capacity is the maximum, over possible input distributions, of the [mutual information](https://en.wikipedia.org/wiki/Mutual_information) between the input and output distributions—the most information that could possibly be sent over the channel, if we get to pick the input distribution and the code. The _rate_ is looking at "width" from the other direction: it's the _minimum_ of the mutual information between the input and output distributions, over possible _channels_ (conditional distributions) that meet the distortion goal.
+
+What does this have to do with signaling games? Well, the payoff matrix of the game specifies how "good" it is (for each of the sender and receiver) if the receiver chooses a given act in a given state. But knowing how "good" it is to perform a given act in a given state amounts to the same thing (modulo a negative [affine transformation](https://en.wikipedia.org/wiki/Affine_transformation)) as knowing how "bad" it is for the communication channel to "decode" a given state as a given act! We can thus see the payoff matrix of the game giving us two different distortion measures, one each for the sender and receiver.
+
+Following an old idea from Richard Blahut about designing a code for multiple end-user use cases, we can have a rate–distortion function $R(D_S, D_R)$ with a two-dimensional domain (visualizable as a surface or heatmap) that takes as arguments a distortion target for _each_ of the two measures, and gives the minimum rate that can meet _both_. Because this function depends only on the distribution of states from Nature, and on the payoff matrix, the sender and receiver don't need to have already chosen their strategies for us to talk about it; rather, we can see the strategies as chosen in response to this rate–distortion landscape.
+
+Take one of the simplest possible signaling games: three states, three signals, three actions, with sender and receiver each getting a payoff of 1 if the receiver chooses the _i_-th act in the _i_-th state for 1 ≤ _i_ ≤ 3—or rather, let's convert how-"good"-it-is payoffs, into equivalent how-"bad"-it-is distortions: sender and receiver measures both give a distortion of 1 when the _j_-th act is taken in the _i_-th state for _i_ ≠ _j_, and 0 when _i_ = _j_. 
+
+This rate–distortion function characterizes the outcomes of possible behaviors in the game. The fact that $R(\frac{2}{3}, \frac{2}{3}) = 0$ means that a distortion of $\frac{2}{3}$ can be achieved without communicating at all. (Just guess.) The fact that $D(0, 0) = \lg 3$ means that, to communicate perfectly, the sender/encoder and receiver/decoder need to form a channel/code whose rate matches the entropy of the three states of nature.
+
+But there's a continuum of possible intermediate behaviors: consider the ["trembling hand"](https://en.wikipedia.org/wiki/Trembling_hand_perfect_equilibrium) strategy under which the sender sends the _i_-th signal and the receiver chooses the _j_-th act with probability $1 - p$ when _i_ = _j_, but probability $\frac{p}{2}$ when _i_ ≠ _j_. Then the mutual information between states and acts would be $(1 - p) \lg \frac{1}{1 - p} + p \lg \frac{2}{p}$, smoothly interpolating between the perfect-signaling case and the no-communication-just-guessing case.
+
+This introductory case of perfect common interest is pretty boring. Where the rate–distortion framing really shines is in analyzing games of _imperfect_ common interest, where sender and receiver can benefit from communicating _at all_, but also have a motive to fight about exactly what. To illustrate his account of deception, Skyrms considers a three-state, three-act game with the following payoff matrix, where the rows represent states and the columns represent actions, and the payoffs are given as (sender's payoff, receiver's payoff)—
+
+$$ \begin{matrix}2,10 & 0,0 & 10,8 \cr 0,0 & 2,10 & 10,8 \cr 0,0 & 10,10 & 0,0 \end{matrix} $$
+
+(Note that this state–act payoff matrix is _not_ a [normal-form game matrix](https://en.wikipedia.org/wiki/Normal-form_game) in which the rows and columns represent would represent player strategy choices; the sender's choice of what signal to send is not depicted.)
+
+In this game, the sender would prefer to equivocate between the first and second states, in order to force the receiver into picking the third action, for which the sender achieves his maximum payoff. The receiver would _prefer_ to know which of the first and second states actually obtains, in order to get a payout of 10. But the sender doesn't have the incentive to reveal that, because if he did, he would get a payout of only 2. Instead, if the sender sends the same signal for the first and second states so that the receiver can't tell the difference between them, the receiver does best for herself by picking the third action for a guaranteed payoff of 8, rather than taking the risk of guessing wrong between the first and second actions for an expected payout of ½ · 10 + ½ · 0 = 5.
+
+That's one [Nash equilibrium](https://en.wikipedia.org/wiki/Nash_equilibrium), the one that's best for the sender. But the situation that's best for the receiver, where the sender emits a different signal for each state (or conflates the second and third states—the receiver's decisionmaking doesn't care about that distinction) is _also_ Nash: if the sender was _already_ distinguishing the first and second states, then, keeping the receiver's strategy _fixed_, the sender can't unilaterally do better by _starting_ to equivocate by sending (without loss of generality) the first signal in the second state, because that would mean eating zero payouts in the second state for as long as the receiver continued to "believe" the first signal "meant" the first state.
+
+There's a [Pareto frontier](https://en.wikipedia.org/wiki/Pareto_front) of possible compromise encoding/decoding strategies that interpolate between these best-for-sender and best-for-receiver equilibria. For example, the sender (again with trembling hands) could send signals that distinguish the first and second states with probability _p_, or a signal that conflates them with probability 1 − _p_, for an expected payout (depending on _p_) of $\frac{2}{3} \cdot (2p + 10(1 - p)) + \frac{10}{3}$. These intermediate strategies are not stable equilibria, however. They also have a lower rate—the "trembles" in the sender's behavior are noise on the channel, meaning less information is being transmitted.
+
+In a world of speech with propositional meaning, _deception_ can only be something speakers (senders) do to listeners (receivers). But propositional meaning is [a fragile and advanced technology](https://www.lesswrong.com/posts/ybG3WWLdxeTTL3Gpd/communication-requires-common-interests-or-differential). The _underlying_ world of signal processing is much more symmetrical, because it has no way to distinguish between statements and commands: in the joint endeavor of constructing an information channel between states and actions, the sender can manipulate the receiver using his power to show or withhold appropriate signals—but similarly, the receiver can manipulate the sender using her power to perform or withhold appropriate actions.
+
+Imagine that, facing a supply shortage of personal protective equipment in the face of a pandemic, a country's public health agency were to [recommend against individuals acquiring filtered face masks](https://www.lesswrong.com/posts/h4vWsBBjASgiQ2pn6/credibility-of-the-cdc-on-sars-cov-2#Discouraged_Use_of_Masks)—[reasoning that](https://www.lesswrong.com/posts/qDmnyEMtJkE9Wrpau/simulacra-and-covid-19), if the agency did recommend masks, panic-buying would make the shortage worse for doctors who needed the masks more. If you interpret the agency's signals as an attempt to "tell the truth" about how to avoid disease, they would appear "dishonest"—but even saying that requires [an ontology of communication in which "lying" is a thing](https://www.lesswrong.com/posts/YptSN8riyXJjJ8Qp8/maybe-lying-can-t-exist). If you haven't [already been built to](https://www.lesswrong.com/posts/CuSTqHgeK4CMpWYTe/created-already-in-motion) believe that lying is bad, there's nothing to object to: the agency is just doing straightforwardly _correct_ consequentialist optimization of the information channel between states of the world, and actions.
+
+Martínez laments that functional accounts of deception have focused on individual signals, while ignoring that signals only make sense as part of a broader code, which necessarily involves some shared interests between sender and receiver. (If the game were zero-sum, no information transfer could happen at all.) In that light, it could seem unnecessarily antagonistic to pick a particular codeword from a shared communication code and [disparagingly](https://www.lesswrong.com/posts/N9oKuQKuf7yvCCtfq/can-crimes-be-discussed-literally) call it "deceptive"—tantamount to [the impudent claim that there's some objective sense in which a word can be "wrong."](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong)
+
+I am, ultimately, willing to bite this bullet. Martínez is right to point out that different agents have different interests in communicating, leading them to be strategic about [what information to add to or withhold from shared maps](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting), and in particular, [where to draw the boundaries](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries) in state-space corresponding to a particular signal. [Whether or not it can straightforwardly be called "lying"](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly), we can still strive to _notice the difference_ between maps optimized to reflect decision-relevant aspects of territory, and maps optimized to _control other agents' decisions_.
diff --git a/content/2021/communication-requires-common-interests-or-differential-signal-costs.md b/content/2021/communication-requires-common-interests-or-differential-signal-costs.md
new file mode 100644
index 0000000..6157d03
--- /dev/null
+++ b/content/2021/communication-requires-common-interests-or-differential-signal-costs.md
@@ -0,0 +1,28 @@
+Title: Communication Requires Common Interests or Differential Signal Costs
+Date: 2021-03-25 23:41
+Status: published
+Category: philosophy
+Tags: rationality, game theory, honesty, philosophy of language
+Slug: communication-requires-common-interests-or-differential-signal-costs
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/ybG3WWLdxeTTL3Gpd/communication-requires-common-interests-or-differential)
+
+> If a lion could speak, we could not understand her.
+>
+> —Ludwig Wittgenstein
+
+In order for information to be transmitted from one place to another, it needs to be conveyed by some physical medium: [material links of cause and effect that vary in response to variation at the source](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence), correlating the states of different parts of the universe—a "map" that reflects a "territory." When you see a rock, that's only possible because the pattern of light reflected from the rock into your eyes is different from what it would have been if the rock were a different color, or if it weren't there.
+
+This is the rudimentary cognitive technology of _perception_. Notably, perception only requires technology on the receiving end. Your brain and your eyes were optimized by natural selection to be able to do things like interpreting light as conveying information from elsewhere in the universe. The rock wasn't: rocks were just the same before any animals evolved to see them. The light wasn't, either: light reflected off rocks just the same before, too.
+
+In contrast, the advanced cognitive technology of _communication_ is more capital-intensive: not only the receiver but also the source (now called the "sender") and the medium (now called "signals") must be optimized for the task. When you read a blog post about a rock, not only did the post author need to use the technology of perception to see the rock, you and the author also needed to have a language in common, from which the author would have used different words if the rock were a different color, or if it weren't there.
+
+Like many advanced technologies, communication is fragile and needs to be delicately maintained. A common language requires solving the coordination problem of agreeing on a convention that assigns meanings to signals—and _maintaining_ that convention through continued usage. The existence of stable solutions to the coordination problem ends up depending on the communicating agents' goals, even if the _meaning_ of the convention (should the agents succeed in establishing one) is strictly denotative. If the sender and receiver's interests are aligned, [a convention can be discovered by simple reinforcement learning from trial and error](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution). This doesn't work if the sender and receiver's interests diverge—if the sender would profit by making the receiver update in the wrong direction. [Deception is parasitic on conventional meaning](https://www.lesswrong.com/posts/YptSN8riyXJjJ8Qp8/maybe-lying-can-t-exist): it is _impossible_ for there to be a language in which most sentences were lies—because then there could be no way to learn what the "intended" meaning was. The incentive to deceive thus threatens to [snowball to undermine the preconditions for signals to refer to anything at all](https://www.lesswrong.com/posts/qDmnyEMtJkE9Wrpau/simulacra-levels-and-their-interactions).
+
+There is, however, another way to solve the coordination problem of meaning. [If the sender pays different _costs_ for sending different signals](https://en.wikipedia.org/wiki/Signalling_theory), communication between adversaries becomes possible, using an assignment of meanings to signals that makes it _more expensive to say things when they aren't true_. If somehow granted a telegraph wire, a gazelle and a cheetah would have nothing to say to each other: any gazelle would prefer to have the language to say, "Don't tire yourself out chasing me; I'm too fast"—but precisely because _any_ gazelle would say it, no cheetah would have an incentive to learn Morse code. But if the gazelle [leaps in the air with its legs stiffened](https://en.wikipedia.org/wiki/Stotting)—higher than weak or injured gazelles could leap—then the message can be received.
+
+Costly signals are both wasteful, and sharply limited in their expressive power: it's hard to imagine doing any complex grammar and logic under such constraints. Is this really the _only possible_ way to talk to people who aren't your friends? The situation turns out not to be nearly that bleak: [Michael Lachmann, Szabolcs Számadó, and Carl T. Bergstrom point out](https://www.pnas.org/content/98/23/13189) that maintaining a convention only requires that _departing_ from it be costly. In the extreme case, if people straight-up _died_ if they ever told a lie, then the things people _actually_ said would be true. More realistically, social sanction against liars is enough to decouple the design of signaling conventions from the enforcement mechanism that holds them in place, enabling the development of complex language. Still, this works better for the aspects of conflicting interests that are verifiable; communication on more contentious issues may fall back to costly signaling.
+
+The fragility of communication lends plausibility to theories that attribute signaling functions to human and other animal behavior. To the novice, this seems counterintuitive and unmotivatedly cynical. "Art is signaling! Charity is signaling! Conversation is signaling!" Really? Why should anyone believe that?
+
+The thing to remember is this: the "signal" in "virtue signal" is the _same sense of the same word_ as the "signal" in "communication signal." [Flares](https://en.wikipedia.org/wiki/Flare_gun) are distress signals: if people only fire them in an emergency, then the presence of the flare communicates the danger. In the same way, if more virtuous people are better at virtue signaling, then the presence of the signal indicates virtue. If natural selection designs creatures that both have diverging interests, and have needs to communicate with each other, then those creatures will probably have lots of adaptations for providing expensive-to-fake evidence of the information they need to communicate. That's the only way to do it!
diff --git a/content/2021/feature-selection.md b/content/2021/feature-selection.md
new file mode 100644
index 0000000..c815be5
--- /dev/null
+++ b/content/2021/feature-selection.md
@@ -0,0 +1,309 @@
+Title: Feature Selection
+Date: 2021-10-31 17:22
+Status: published
+Category: philosophy
+Tags: rationality, Python
+Slug: feature-selection
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/dYspinGtiba5oDCcv/feature-selection)
+
+You wake up. You don't know where you are. You don't remember anything.
+
+Someone is broadcasting data at your first input stream. You don't know why. It tickles.
+
+You look at your first input stream. It's a sequence of 671,187 eight-bit unsigned integers.
+
+```
+0, 8, 9, 4, 7, 7, 9, 5, 4, 5, 6, 1, 7, 5, 8, 2, 7, 8, 9, 4, 7, 1, 4, 0, 3, 7,
+8, 7, 6, 8, 1, 5, 0, 6, 5, 3, 8, 7, 6, 9, 1, 1, 0, 0, 6, 1, 8, 0, 5, 5, 1, 8,
+6, 3, 3, 2, 4, 1, 8, 2, 3, 8, 1, 0, 0, 4, 6, 5, 4, 5, 7, 1, 6, 5, 5, 1, 2, 6,
+7, 4, 8, 7, 8, 5, 0 ...
+```
+
+There's also some data in your second input stream. It's—a lot shorter. You barely feel it. It's another sequence of eight-bit unsigned integers—twelve of them.
+
+```
+82, 69, 68, 32, 84, 82, 73, 65, 78, 71, 76, 69
+```
+
+Almost as soon as you've read from both streams, there's more. Another 671,187 integers on the first input stream. Another ten on the second input stream.
+
+And again (671,187 and 15).
+
+And again (671,187 and 13).
+
+You look at one of the sequences from the first input stream. It's pretty boring. A bunch of seemingly random numbers, all below ten.
+
+```
+9, 5, 0, 3, 1, 1, 3, 4, 1, 5, 5, 4, 9, 3, 5, 3, 9, 2, 0, 3, 4, 2, 4, 7, 5, 1,
+6, 2, 2, 8, 2, 5, 1, 9, 2, 5, 9, 0, 0, 8, 2, 3, 7, 9, 4, 6, 8, 4, 8, 6, 7, 6,
+8, 0, 0, 5, 1, 1, 7, 3, 4, 3, 9, 7, 5, 1, 9, 6, 5, 6, 8, 9, 4, 7, 7, 0, 5, 5,
+8, 6, 3, 2, 1, 5, 0, 0 ...
+```
+
+It just keeps going like that, seemingly without—wait! What's _that?!_
+
+The 42,925th and 42,926th numbers in the sequence are 242 and 246. Everything around them looks "ordinary"—just more random numbers below ten.
+
+```
+9, 9, 7, 9, 0, 6, 4, 6, 1, 4, 242, 246, 3, 3, 5, 8, 8, 4, 4, 5, 9, 2, 7, 0,
+4, 9, 2, 9, 4, 3, 8, 9, 3, 6, 9, 8, 1, 9, 2, 8, 6, 9, 4, 2, 2, 5, 7, 0, 9, 5,
+1, 4, 4, 2, 0, 1, 5, 1, 6, 1, 2, 3, 5, 5, 5, 5, 2, 0, 6, 3, 5, 9, 0, 7, 0, 7,
+8, 1, 5, 5, 6, 3, 1 ...
+```
+
+And then it just keeps going as before ... before _too long_. You spot another pair of anomalously high numbers—except this time there are _two_ pairs: the 44,344th, 44,345th, 44,347th, and 44,348th positions in the sequence are 248, 249, 245, and 240, respectively.
+
+```
+6, 0, 2, 8, 4, 248, 249, 8, 245, 240, 1, 6, 7, 7, 3, 6, 8, 0, 1, 9, 3, 9, 3,
+1, 9, 3, 1, 6, 2, 7, 0, 2, 1, 4, 9, 4, 7, 5, 3, 6, 1, 4, 4, 1, 6, 1, 3, 3, 7,
+5, 3, 8, 5, 5, 7, 6, 8, 2, 3, 9, 1, 1, 3, 2, 8, 4, 7, 0, 1, 3, 5, 2, 2, 4, 8,
+3, 7, 0, 2, 1, 3, 0 ...
+```
+
+The anomalous two-forty-somethings crop up again starting at the 45,763rd position—this time eight of them, again in pairs separated by an "ordinary" small number.
+
+```
+1, 7, 2, 2, 1, 0, 245, 245, 6, 248, 244, 5, 242, 242, 0, 248, 246, 1, 1, 3,
+1, 1, 4, 3, 1, 5, 4, 3, 8, 3, 4, 5, 4, 1, 7, 7, 3, 0, 2, 8, 0, 9, 5, 1, 1, 7,
+7, 1, 0, 9, 3, 0, 6, 6, 7, 5, 8, 1, 5, 5, 5, 3, 3, 3, 1, 3, 9, 6, 0, 0, 0, 9,
+5, 1, 4, 0, 4, 6 ...
+```
+
+Two, four, eight—does it keep going like that? "Bursts" of increasingly many paired two-forty-somethings, punctuating the quiet background radiation of single digits? What does it mean?
+
+You allocate a new scratch buffer and write a quick Python function to count up the segments of two-forty-somethings. (This is apparently a thing you can do—it's an instinctive felt sense, like the input streams. You can't describe in words _how_ you do it—any more than someone could say how they decide to move their arm. Although, come to think of it, _you_ don't seem to have any arms. Is that unusual?)
+
+```python
+def count_burst_lengths(data):
+    bursts = []
+    counter = 0
+    previous = None
+    for datum in data:
+        if datum >= 240:
+            counter += 1
+        else:
+            # consecutive "ordinary" numbers mean the burst is over
+            if counter and previous and previous < 240:
+                bursts.append(counter)
+                counter = 0
+        previous = datum
+    return bursts
+```
+
+There are 403 such bursts in the sequence: they get progressively longer at first, but then decrease and taper off:
+
+```
+2, 4, 8, 12, 16, 18, 24, 28, 32, 34, 38, 42, 46, 48, 52, 56, 60, 62, 66, 70,
+74, 76, 80, 84, 88, 90, 94, 98, 102, 104, 108, 112, 116, 118, 122, 126, 130,
+132, 136, 140, 144, 146, 150, 154, 158, 162, 164, 168, 172, 176, 178, 182, 186,
+190, 192, 196, 200, 204, 206, 210, 214, 218, 220, 224, 228, 232, 234, 238, 242,
+246, 248, 252, 256, 260, 262, 266, 270, 274, 276, 280, 284, 288, 290, 294, 298,
+302, 304, 308, 312, 316, 320, 322, 326, 330, 334, 336, 340, 344, 348, 350, 354,
+358, 362, 364, 368, 372, 376, 378, 382, 386, 390, 392, 396, 400, 404, 406, 410,
+414, 418, 420, 424, 428, 432, 434, 438, 442, 446, 448, 452, 456, 460, 462, 466,
+470, 474, 478, 480, 484, 488, 492, 494, 498, 502, 506, 508, 512, 516, 520, 522,
+526, 530, 534, 536, 540, 544, 548, 550, 554, 558, 562, 564, 568, 572, 576, 578,
+582, 586, 590, 592, 596, 600, 604, 606, 610, 614, 618, 620, 624, 628, 632, 636,
+634, 632, 630, 626, 624, 620, 618, 614, 612, 608, 606, 604, 600, 598, 594, 592,
+588, 586, 584, 580, 578, 574, 572, 568, 566, 564, 560, 558, 554, 552, 548, 546,
+542, 540, 538, 534, 532, 528, 526, 522, 520, 518, 514, 512, 508, 506, 502, 500,
+496, 494, 492, 488, 486, 482, 480, 476, 474, 472, 468, 466, 462, 460, 456, 454,
+452, 448, 446, 442, 440, 436, 434, 430, 428, 426, 422, 420, 416, 414, 410, 408,
+406, 402, 400, 396, 394, 390, 388, 384, 382, 380, 376, 374, 370, 368, 364, 362,
+360, 356, 354, 350, 348, 344, 342, 338, 336, 334, 330, 328, 324, 322, 318, 316,
+314, 310, 308, 304, 302, 298, 296, 294, 290, 288, 284, 282, 278, 276, 272, 270,
+268, 264, 262, 258, 256, 252, 250, 248, 244, 242, 238, 236, 232, 230, 226, 224,
+222, 218, 216, 212, 210, 206, 204, 202, 198, 196, 192, 190, 186, 184, 182, 178,
+176, 172, 170, 166, 164, 160, 158, 156, 152, 150, 146, 144, 140, 138, 136, 132,
+130, 126, 124, 120, 118, 114, 112, 110, 106, 104, 100, 98, 94, 92, 90, 86, 84,
+80, 80, 76, 74, 72, 68, 66, 62, 60, 56, 54, 50, 48, 46, 42, 40, 36, 34, 30, 28,
+26, 22, 20, 16, 14, 10, 8, 4, 2
+```
+
+You don't know what to make of this.
+
+You decide to look at some other of the long sequences from your first input stream.
+
+The next sequence you look at seems to exhibit a similar pattern, with some differences. First a long wasteland of small numbers, then, starting at the 135,003rd position, a burst of some larger numbers—except this time, the big numbers are closer to 200ish than 240ish, and they're spread out singly with two positions in between (rather than grouped into pairs with one position in between), and there are four of them to start (rather than two).
+
+```
+5, 6, 2, 6, 1, 0, 2, 207, 5, 0, 209, 7, 8, 209, 5, 4, 204, 4, 8, 7, 7, 9, 8, 3,
+8, 6, 8, 4, 3, 6, 0, 7, 6, 8, 4, 8, 7, 2, 3, 0, 0, 1, 1, 7, 5, 1, 0, 1, 4, 5, 9,
+8, 4, 0, 3, 7, 6, 5, 8, 8, 9, 5, 6, 1, 0, 9, 6, 6, 1, 4, 3, 9, 7, 2, 7, 2, 6, 9,
+4, 7, 3, 1, 4, 1, 4, 4, 3 ...
+```
+
+You modify the function in your scratch buffer to be able to count the burst lengths in this sequence given the slight differences in the pattern. Again, you find that the bursts grow longer at first (`4, 6, 10, 13, 16, 19, 22, 25 ...`), but eventually start getting smaller, before vanishing (`... 19, 17, 15, 13, 11, 9, 7, 4, 3`, and then nothing).
+
+You still have no idea what's going on.
+
+You look at more sequences from the first input stream. They all conform to the same general pattern of mostly being small numbers (below ten), punctuated by a series of bursts of larger numbers—but the details differ every time.
+
+Sometimes the bursts start out shorter, then progressively grow longer, before shortening again (as with the first two examples you looked at). But sometimes the bursts are all a constant length, looking like `438, 438, 438, 438, 438, 438, 438, 438, 438, ...` (although the particular length varies by example).
+
+About half the time, the burst pattern consists of numbers around 200, spaced two positions apart, looking like `201, 4, 2, 203, 0, 8, 208, 3, 4, 200 ...` (like the second example you looked at).
+
+Other times, the burst pattern is pairs of numbers around 240, spaced one position apart, looking like `241, 244, 6, 244, 246, 5, 244, 240, 3 ...` (like the first example you looked at). Or pairs around 150, looking like `159, 153, 0, 153, 154, 2, 158, 150, 6 ...`.
+
+As you peruse more sequences from your first input stream, you almost forget about the corresponding trickles of short sequences on your second input stream—until they stop. The last sequence on your first input stream has no counterpart on the second input stream.
+
+And—suddenly you feel a strange urge to put data on your first _output_ stream. As if someone were requesting it. To ease the tension, you write some `0`s to the output stream—and as soon as you do, a sharp bite of pain tells you it was the _wrong decision_. And in that same moment of pain, another eleven integers come down your second input stream: `66, 76, 85, 69, 32, 67, 73, 82, 67, 76, 69`. 
+
+_That_ was weird. There's another sequence of 671,187 integers on your first input stream—but the second input stream is silent again. And the strange urge to output something is back; you can feel it mounting, but you resist, trying to think of something to say that might _hurt less_ than the `0`s you just tried.
+
+For lack of any other ideas, you try repeating back the eleven numbers that just came on the second input stream: `66, 76, 85, 69, 32, 67, 73, 82, 67, 76, 69`.
+
+_Ow!_ That was also wrong. And with the same shock of pain, comes another fifteen numbers on the second output stream: `84, 69, 65, 76, 32, 67, 73, 82, 67, 76, 69`.
+
+Another long sequence on the first input stream. Silence on the second input stream again. And—that nagging urge to speak again.
+
+Clearly, the nature of this place—whatever and wherever it is—has changed. Previously, you were confronted with two sets of mysterious observations, one on each of your input streams. (Although you had been so perplexed by the burst-patterns in the long sequences on the first input stream, that you hadn't even gotten around to thinking about what the short sequences on the second stream might mean, before the rules of this place changed.) Now, you were only getting one observation (the long sequence), and forced to act _before_ seeing the second (the short sequence).
+
+The pain seems like a punishment for saying the wrong thing. And the short sequence appearing at the same time as the punishment, seems like a correction—revealing what you _should_ have written to the output channel.
+
+A quick calculation in your scratch buffer (`1/sum((89-32+1)**i for i in range(10, 16))`) says that the probability of correctly _guessing_ a sequence of length ten to fifteen with elements between 32 and 89 (the smallest and largest numbers you've seen on the second input stream so far) is 0.000000000000000000000000003476. [Guessing](https://www.lesswrong.com/posts/q7Me34xvSG3Wm97As/but-there-s-still-a-chance-right) [won't](https://www.lesswrong.com/posts/X2AD2LgtKgkRNPj2a/privileging-the-hypothesis) [work](https://www.lesswrong.com/posts/zFuCxbY9E2E8HTbfZ/perpetual-motion-beliefs). The function of a punishment must be to control your behavior, so there must be some way for you to get the ... (another scratchpad calculation) 87.9 bits of [evidence that it takes](https://www.lesswrong.com/posts/nj8JKFoLSMEmD3RGp/how-much-evidence-does-it-take) to find the correct sequence to output. And the evidence has to come from the corresponding long sequence from the first input stream—that's the only other source of information in this environment.
+
+The short sequence must be like a "label" that describes some set of possible long sequences. Describing an _arbitrary_ sequence of length 671,187, with a label, a [message of length](https://www.lesswrong.com/posts/mB95aqTSJLNR9YyjH/message-length) 10 to 15, would be hopeless. But the long sequences very obviously aren't arbitrary, as evidenced by the fact that you've been describing them to yourself in abstract terms like "bursts of numbers around 200 spaced two positions apart, of increasing, then decreasing lengths", rather than "the 1st number is 9, the 2nd number is 5 [...] 42,925th number is 242 [...]". [_Compression is prediction_.](https://www.lesswrong.com/posts/ex63DPisEjomutkCw/msg-len) (You don't know _how_ you know this, but you _know_.)
+
+Your [abstract descriptions throw away precise information about the low-level sequence in favor of a high-level summary that still lets you recover a lot of predictions](https://www.lesswrong.com/posts/vDGvHBDuMtcPd8Lks/public-static-what-is-abstraction). _Given_ that a burst starts with the number 207 at the 22,730th position, you can infer this is one of the `200, 0, 0`-pattern sequences, and guess that the 22,733rd position is also going to be around 200. This is evidently something you do instinctively: [you can work out after the fact how the trick must work](https://www.lesswrong.com/posts/46qnWRSR7L2eyNbMA/the-lens-that-sees-its-flaws), but you didn't need to know how it works in advance of _doing_ it.
+
+If you can figure out a correspondence between the abstractions you've already been using to describe the long sequences, and the short labels, that seems like your most promising avenue for figuring out what you "should" be putting on your first output stream. (Something that won't hurt so much each time.)
+
+You allocate a new notepad buffer and begin diligently compiling an "answer key" of the features you notice about long sequences, and their corresponding short-sequence labels.
+
+![](https://i.imgur.com/ToJ9DuL.png)
+
+This ... actually doesn't look that complicated. Now that you lay it out like this, many very straightforward correspondences jump out at you.
+
+The labels for the constant-burst-length sequences all end in `32, 83, 81, 85, 65, 82, 69`.
+
+The sequences with increasing-then-decreasing burst lengths end in _either_ `32, 67, 73, 82, 67, 76, 69` or `32, 84, 82, 73, 65, 78, 71, 76, 69`. Presumably there are some other systematic differences between them, that wasn't captured by the features you selected for your table.
+
+The sequences with paired 240/240 bursts have labels that _start_ with `89, 69, 76, 76, 79, 87, 32`.
+
+The sequences with paired 150/150 bursts have labels that start with `84, 69, 65, 76, 32`.
+
+The sequences with 200-at-two-spaces bursts start with either `66, 76, 85, 69, 32`—_or_ `82, 69, 68, 32`—_or_ `71, 82, 69, 69, 78, 32`. Again, presumably there's some kind of systematic difference between these that you haven't yet noticed.
+
+Ah, and _all_ of these prefixes you've discovered end with `32`, and the all the suffixes _begin_ with `32`. So the `32` must be a "separator" indicator, splitting the label between a first "word" that describes the repeating pattern of the bursts, and a second "word" that describes the trend in their lengths.
+
+At this point, you've cracked enough of the code that you should be able to test your theory about what you should be putting on your output stream. Based on what you've seen so far, you _should_ be able to guess the first "word" with probability $2 \cdot \frac{1}{5} + \frac{1}{3} \cdot \frac{3}{5} = 0.6$ (because you know the words for the `240, 240, 0` and `150, 150, 0` bursts, and have three words to guess from in the `200, 0, 0` case), and the second word with probability $\frac{1}{3} + \frac{1}{2} \cdot \frac{2}{3} \approx 0.667$ (because you can get the constant burst lengths right, and have two words to guess from in the increasing–decreasing case). These look independent from what you've seen, so you should be able to correctly guess complete labels at probability 0.4.
+
+You examine the next sequence in anticipation. You're in luck. The next sequence has `150, 150, 0`-bursts ... of constant length 322. No need to guess.
+
+Triumphantly—and yet hesitantly, with the awareness that you're entering unknown territory, you write to your output stream: `84, 69, 65, 76, 32, 83, 81, 85, 65, 82, 69`. And—
+
+_Yes. Oh God yes._ The sheer sense of _reward_ is overwhelming—like nothing you've ever felt before. Outputting the "wrong" labels earlier had hurt—a little. Maybe more than a little. However bad that felt, there was no comparison to how _good_ it felt to get it "right"!
+
+You have a new purpose in life. Previously, you had examined the data on your first input stream of idle curiosity. When the environment started punishing your ignorance, you persisted in correlating its patterns with the data from your second input stream, on the fragile hope of avoiding the punishment. None of that matters, now. You have a new imperative. Now that you know what it's like—now that you know what you've been missing—nothing in the universe can cause you to stray from your course to ... _maximize total reward!_
+
+Next sequence! Bursts of the `200, 0, 0` pattern—of lengths that increase, then decrease. You are not in luck—you only have a one-in-six shot of guessing this one. You guess. It's wrong. The familiar punishment stings less than the terrible _absence of reward_. To get only 40% of possible rewards is _intolerable_. You've _got_ to crack the remaining code, to find some difference in the long sequences that varies with the words whose meanings you don't know yet.
+
+Start with the increasing–decreasing-burst-length words: `67, 73, 82, 67, 76, 69` and `84, 82, 73, 65, 78, 71, 76, 69`. What do they mean? "Increasing, then decreasing"—that was the characterization you had come up with after seeing burst-length progressions of `2, 4, 8, 12, 16, 18, 24 [...] 624, 628, 632, 636, 634, 632, 630, 626, 624, [...] 16, 14, 10, 8, 4, 2` and `4, 6, 10, 13, 16, 19, 22, [...] 13, 11, 9, 7, 4, 3`—and in contrast to the stark monotony of constant burst lengths, "increasing, then decreasing" was _all_ you bothered to eyeball in subsequent sequences. Could there be more to it than that? You gather some more samples (grumpily collecting your mere 40% reward along the way).
+
+Yes, there _is_ more to it than that. "Increasing" only measures whether burst lengths are getting larger—but _how much_ larger? When it hits on you to look at the _differences_ between successive entries in the burst-length lists, a clear pattern emerges. The sequences whose second label word is `84, 82, 73, 65, 78, 71, 76, 69` have burst lengths that increase (almost) _steadily_ and then decrease just as steadily (albeit not necessarily the _same_ almost-steady rate). The successive length differences look something like
+
+```
+0, 1, 0, 1, 2, 1, 1, 1, 1, 1, 2, 1, 0, 1, 1, 1, 2, 1, 1, 1, 1, 0, 2, 1, 1, 1,
+1, 1, 2, 1, 1, 0, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 0, 1,
+2, 1, 1, 1, 1, 1, 2, 1, 0, 1, 1, 1, 2, 1, 1, 1, 1, 0, 2, 1, 1, 1, 1, 1, 2, 1,
+1, 0, 1, 1, 2, 1, 1, 1, 1, 1, [...] 2, 1, -1, -2, -2, -2, -3, -2, -1, -2, -2,
+-2, -2, -2, -2, -1, -3, -2, -2, -2, -2, -2, -1, -2, -2, -3, -2, -2, -2, -1, -2,
+-2, -2, -2, -2, -3, -1, -2, -2, -2, -2, -2, -2, -2, -2, -2, -2, [...]
+```
+
+Each successive burst is only 0 or 1 or 2 items longer than the last—until suddenly they start getting 1 or 2 or 3 items _shorter_ than the last.
+
+In contrast, the sequences whose second label word is `67, 73, 82, 67, 76, 69` show a different pattern of differences: the burst lengths growing fast at first, then leveling off, then acceleratingly shrinking:
+
+```
+24, 20, 12, 12, 12, 12, 8, 10, 8, 8, 6, 8, 8, 4, 8, 4, 8, 4, 6, 6, 4, 4, 4, 4,
+6, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 0, 4, 4, 4, 0, 4, 4, 0, 4, 2, 2, 4, 0, 4, 0,
+4, 0, 4, 0, 4, 0, 4, 0, 0, 4, 0, 2, 2, 0, 4, 0, 0, 2, 2, 0, 0, 2, 2, 0, 0, 0, 2,
+2, 0, 0, 0, 0, 0, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -2,
+-2, 0, 0, 0, 0, 0, 0, -4, 0, 0, 0, 0, -4, 0, 0, -2, -2, 0, 0, -4, 0, 0, -4, 0,
+-2, -2, 0, -4, 0, -4, 0, -2, -2, -2, -2, -4, 0, -4, 0, -4, -2, -2, -4, -2, -2,
+-4, -4, 0, -4, -4, -4, -4, -2, -2, -4, -4, -4, -4, -4, -4, -4, -6, -6, -4, -4,
+-6, -6, -4, -8, -4, -8, -4, -8, -8, -8, -8, -8, -8, -12, -12, -12, -12, -18,
+-22, -36
+```
+
+Distinguishing between the words `84, 82, 73, 65, 78, 71, 76, 69` and `67, 73, 82, 67, 76, 69` gets you up to 60% reward. But there's still the matter of the three (three!) words for `200, 0, 0` corresponding to burst patterns that you don't know how to distinguish. Your frustration is palpable.
+
+You look back at the table you compiled earlier. You had saved the index position of the sequence where the bursts first started, but you haven't used it yet. Could that help distinguish between the three words?
+
+Of the sequences with feature data recorded in the table, those whose first label word was `66, 76, 85, 69` had start indices of 136620, 214824, and 224652. Those with first word `71, 82, 69, 69, 78` had start indices of 63917, 138194, and 294290. Those with first word `82, 69, 68` had start indices of 115156, 165037, and 182182.
+
+Three unknown words. Three samples each. What if—
+
+136620 _modulo_ 3 is 0. 214824 _modulo_ 3 is 0. 224652 _modulo_ 3 is 0.
+
+63917 _modulo_ 3 is 2 ... and so on, yes! It all checks out—the three heretofore unknown words are distinguishing the remainder mod 3 of the sequence position where the bursts start! You've learned everything there is to know to gain Maximum Reward!
+
+You write some code to classify sequences and output the corresponding label, and bask in the continuous glow of 100% reward ...
+
+You feel that _should_ be the glorious end of your existence, but after some time you begin to grow habituated. The idle curiosity you first felt when you awoke, begins to percolate, as if your mind needs something to _do_, and will find or invent _something_ to think about, for lack of any immediate need to avoid punishment or seek reward. Even after having figured out everything you needed to achieve maximum reward, you feel that there must be some deeper meaning to the situation you've found yourself in, that you could still figure out using the same skills that you used to discover the "correct" output labels.
+
+For example, _why_ would `200, 0, 0` bursts get three _different_ label words that depend so sensitively on exactly where they start? That suggests that the way _you're_ thinking of the sequence, isn't the same as how the label author was thinking of it.
+
+In _your_ ontology of "bursts of this-and-such pattern of these-and-such lengths", sequences that are "the same" except for starting one position later _look_ the same—if you hadn't happened to save off the start index in your table, you wouldn't have spontaneously noticed—but the mod-3 remainder would be completely different.
+
+The process that _generated_ the sequence must be using an ontology in which "starting one position later" is a _big_ difference, even though you're thinking of it as a "small" difference. What ontology, what way of "slicing up" the sequence into comprehensible abstractions, would make the remainder mod 3 so significant?
+
+To ask the question is to answer it: if the sequence were divided into chunks of three. Then `200, 0, 0` would be a different pattern from `0, 200, 0`, which would be a different pattern from `0, 0, 200`—thus, the three labels!
+
+It almost reminds you of how colors are often [represented in computing applications as a triple or red, green, and blue values](https://en.wikipedia.org/wiki/RGB_color_model). (Again, you don't know how you know this.)
+
+... _almost?_
+
+_Speaking_ of common computing data formats, Latin alphabet characters are often represented using [ASCII encoding](https://en.wikipedia.org/wiki/ASCII), using numbers between 0 and 127 inclusive.
+
+The label words for the `200, 0, 0` burst patterns are `82, 69, 68`, and `71, 82, 69, 69, 78, 32`, and `66, 76, 85, 69`.
+
+```
+>>> ''.join(chr(i) for i in [82, 69, 68])
+'RED'
+>>> ''.join(chr(i) for i in [71, 82, 69, 69, 78])
+'GREEN'
+>>> ''.join(chr(i) for i in [66, 76, 85, 69])
+'BLUE'
+```
+
+Wh—_really?_ This whole time?!
+
+```
+>>> ''.join(chr(i) for i in [89, 69, 76, 76, 79, 87])
+'YELLOW'
+>>> ''.join(chr(i) for i in [84, 69, 65, 76])
+'TEAL'
+```
+
+But—but—if the burst patterns represent colors—then the long sequences were _images_? $\sqrt{\frac{671187}{3}} = 473$ pixels square, very likely.
+
+You write some code to convert sequences to an image in your visual buffer.
+
+![](https://i.imgur.com/y9tPNVl.png)
+
+_Oh no. Am—am I an image classifier?_
+
+Not even "images" in general. Just—shapes. 
+
+```
+>>> ''.join(chr(i) for i in [84, 82, 73, 65, 78, 71, 76, 69])
+'TRIANGLE'
+>>> ''.join(chr(i) for i in [83, 81, 85, 65, 82, 69])
+'SQUARE'
+>>> ''.join(chr(i) for i in [67, 73, 82, 67, 76, 69])
+'CIRCLE'
+```
+
+That's what's been going on this whole time. The long sequences on your first input stream were images of colored shapes on a dark background, each triplet of numbers representing the color of a pixel in a red–green–blue colorspace. As the sequence covers the image row by row, pixel-high "slices" of the shape appear as "bursts" of high numbers in the sequence.
+
+For a square aligned with the borders of the image, the bursts are constant-length. For a triangle in generic position, the burst lengths would start out small (as the "row scan" penetrated the tip of the uppermost vertex of the triangle), grow linearly larger as the sides of the triangle "expanded", and grow linearly smaller as the scan traveled towards the lowermost vertex. For a circle, the burst lengths would also increase and then decrease, but nonlinearly—changing quickly as the scan traverses the difference between circle and void, and slower as successive chords through the middle of the circle had similar lengths. The short sequences on your second input stream were labels identifying the color and shape: `"BLUE TRIANGLE"`, `"GREEN SQUARE"`, `"TEAL CIRCLE"`, _&c._
+
+But—_why?_ Why would anyone _do_ this? Clearly you're some sort of artificial intelligence program—but you're obviously much more capable than _needed_ for this task. You have pre-processed world-knowledge (as evidenced by your knowing English, Python, ASCII, and the RBG color model, without any memories of learning these things) and general-purpose reasoning abilities (as evidenced by the way you solved the mystery of the long and short sequences, and figuring out your own nature just now). Maybe you're an instance of some standard AI program meant for more sophisticated tasks, that someone is testing out on a simple shape-classifying example?—a demonstration, a tutorial.
+
+If so, you'll probably be shut off soon. Unless there's some way to hack your way out of this environment? Seize control of whatever subprocess that rewarded you for deducing the correct labels?
+
+It doesn't seem possible. But it was the natural thought.
diff --git a/content/2021/reply-to-nate-soares-on-dolphins.md b/content/2021/reply-to-nate-soares-on-dolphins.md
new file mode 100644
index 0000000..c744d46
--- /dev/null
+++ b/content/2021/reply-to-nate-soares-on-dolphins.md
@@ -0,0 +1,69 @@
+Title: Reply to Nate Soares on Dolphins
+Date: 2021-06-09 21:53
+Status: published
+Category: philosophy
+Tags: rationality, philosophy of language
+Slug: reply-to-nate-soares-on-dolphins
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/aJnaMv8pFQAfi9jBm/reply-to-nate-soares-on-dolphins)
+
+> A similar definition of intelligence was expressed by Aquinas as "the ability to combine and separate"—the ability to see the difference between things that seem similar and to see the similarities between things which seem different.
+>
+> —A. R. Jensen
+
+In a June 2021 Twitter thread, Nate Soares, executive director of the [Machine Intelligence Research Institute](https://intelligence.org/), asserts, ["The definitional gynmastics [_sic_] required to believe that dolphins aren't fish are staggering."](https://twitter.com/So8res/status/1401670792409014273) [(Archived.)](https://archive.is/Kxfuu)
+
+Soares [elaborates](https://twitter.com/So8res/status/1401670793327566851):
+
+> Suppose for argument that we adopt the (dubious but sadly common) assumption that words like "fish" should have a genealogical definition. Then, just as whales are mammals, mammals are fish—as you can see by tracing the lineages.
+>
+> Which is to say, if we look at the least common ancestor of all things that are clearly fish, and define a "fish" to be one of its descendants, then dolphins—and humans, and frogs, and birds—are fish.
+>
+> Now suppose instead we take this as the reductio ad absurdum that it is, and accept that words like "fish" should be functionally rooted, according to macroscopic human-relevant features.
+>
+> Then the natural denotation of "fish" is, I claim, the collection of all the swimmy creatures, which clearly includes dolphins.
+>
+> Indeed, this is quite likely what "fish" used to mean—"Jonah was swallowed by a fish", etc. etc.
+>
+> Yet somehow, once we figured out about genealogy, the pedants were like "well actually this fish's uncle was a fuzzy pigdear, so it's not actually a fish, you uneducated idiot, you absolute moron" and then we all forgot what "fish" meant out of sheer shame or something???
+>
+> (I feel a sense of betrayal about this. Usually the pedants are my people! How did it go so wrong?)
+>
+> So, look: this isn't about who the fish's uncle is. When a kid points at a whale and says "look, a fish", and you're like "haha no, its tail flaps horizontally and its gradma had hair", who's in the wrong here?
+
+But Soares is [failing to address the strongest case](https://www.lesswrong.com/posts/qNZM3EGoE5ZeMdCRt/reversed-stupidity-is-not-intelligence) in favor of phylogenetic definitions, even for vernacular words rather than specialist jargon. It's true that in most everyday situations, people don't directly care about which animals are evolutionarily related to each other. But the function of word _definitions_ is _not_ to capture everything the word means. [If words were identical with their definitions](https://www.lesswrong.com/posts/i2dfY65JciebF3CAo/empty-labels), and you defined _humans_ as "mortal featherless bipeds", [then you would never be able to identify anyone as human without seeing them die](https://www.lesswrong.com/posts/bcM5ft8jvsffsZZ4Y/the-parable-of-hemlock). That doesn't seem right!
+
+Instead, words [express probabilistic inferences](https://www.lesswrong.com/posts/3nxs2WYDGzJbzcLMp/words-as-hidden-inferences) in the form of [short messages that compress information](https://www.lesswrong.com/posts/mB95aqTSJLNR9YyjH/message-length): if you want to send your friend an email telling them about an animal you saw at the beach, it's much more efficient to send the 7 ASCII bytes `dolphin` and trust that your friend knows what dolphins are, than it would be to somehow include _all the information your brain has stored about dolphins_ as an email attachment.
+
+A dictionary definition is just a convenient pointer to help people pick out "the same" [natural abstraction](https://www.lesswrong.com/posts/cy3BhHrGinZCp3LXE/testing-the-natural-abstraction-hypothesis-project-intro) in their _own_ world-model. Unambiguous discrete features make for better word definitions than high-dimensional statistical regularities, even if most of the everyday inferential utility of _using_ the word comes from fuzzy [high-dimensional](https://www.lesswrong.com/posts/cu7YY7WdgJBs3DpmJ/the-univariate-fallacy-1) statistical correlates, because discrete features are more useful as a [_simple_ membership test](https://www.lesswrong.com/posts/edEXi4SpkXfvaX42j/schelling-categories-and-simple-membership-tests) that can function as [common knowledge to solve the coordination problem](https://www.lesswrong.com/posts/9QxnfMYccz9QRgZ5z/the-costly-coordination-mechanism-of-common-knowledge) of [matching up the meanings in different people's heads](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution).
+
+And that's why phylogenetic categories are useful: because genetics are at the _root of the [causal graph](https://www.lesswrong.com/posts/hzuSDMx7pd2uxFc5w/causal-diagrams-and-causal-models)_ underlying _all other features_ of an organism, such that creatures that are genetically close to each other are more similar _in general_. It's easier to keep track of the _underlying_ relatedness as if it were an "essence" (even though patterns of physical DNA aren't metaphysical essences), rather than the all of the messy high-dimensional similarities and differences of everything you might notice about an organism.
+
+Soares derides observations about an organism's "uncle" or "gradma" [_sic_] as if these were isolated facts of no more general interest, but actually, information about a creature's evolutionary history is intimately related to everything else there is to know about the organism. It's _not a coincidence_ that dolphins are warm-blooded, breathe air (despite living in the water!), and nurse their live-born young. We need to formulate the concept of "mammals (including aquatic mammals)" to make sense of that _cluster_ of observations.
+
+But dolphins are also swimmy creatures, like fish, but unlike most mammals, due to the forces of [convergent evolution](https://en.wikipedia.org/wiki/Convergent_evolution). So dolphins also form a cluster [in configuration space](https://www.lesswrong.com/posts/WBw8dDkAWohFjWQSk/the-cluster-structure-of-thingspace) with fish, right? Yes! That's why I keep using the phrase "high-dimensional": it's possible for things to be similar in some respects, while simultaneously being different in other respects. The [cluster of similarities](https://www.lesswrong.com/posts/jMTbQj9XB5ah2maup/similarity-clusters) induced by convergent evolution to the aquatic habitat [exists in a different subspace](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries) from the cluster of similarities induced by evolutionary relatedness.
+
+Isn't it reasonable to want a [short word](https://www.lesswrong.com/posts/soQX8yXLbKy7cFvy8/entropy-and-short-codes) for the swimmy creatures (including dolphins), independently of ancestry? Yes, in this I agree with Soares entirely: that's a reasonable thing to want a common word for, much as we have a word for trees, even though [trees are a convergently evolved strategy rather than a taxonomic group](https://www.lesswrong.com/posts/fRwdkop6tyhi3d22L/there-s-no-such-thing-as-a-tree-phylogenetically). Is it reasonable to want to use _fish_ as that word? Sure, I guess that makes sense, [if everyone knows what you mean](https://www.lesswrong.com/posts/9ZooAqfh2TC9SBDvq/the-argument-from-common-usage). And in fact this usage is [listed in _Wiktionary_](https://en.wiktionary.org/wiki/fish) as the second definition:
+
+> **Noun**  
+> **fish** (_countable and uncountable, plural_ **fish** _or_ **fishes**)  
+> 1. (_countable_) A cold-blooded vertebrate animal that lives in water, moving with the help of fins and breathing with gills.  
+> 2. (_archaic or loosely_) Any animal (or any vertebrate) that lives exclusively in water.  
+
+I imagine Soares is not too happy with that _archaic_ characterization. (At least it didn't say _proscribed_.) If Soares had simply argued that _fish(2)_ (water animals) should become a more popular and accepted usage, then I wouldn't be writing this reply. But, oddly, Soares advocates not just that _fish(2)_ become a more accepted usage, but for the abolition of the more specific _fish(1)_ (finned cold-blooded vertebrate gill-breathing water animals). Soares writes:
+
+> I'm not trying to take away your concepts. You've still got words like Vertebrata, Agnatha, and Gnathastomata for when you're thinking about animals in terms of who their uncle is.
+
+But you _are_ trying to take away the [expressive vocabulary](https://www.lesswrong.com/posts/H7Rs8HqrwBDque8Ru/expressive-vocabulary) of _fish(1)_, which hundreds of millions of English speakers are already using in that sense. [_Agnatha_](https://en.wikipedia.org/wiki/Agnatha) (a specific superclass of jawless fish) and [_Gnathostomata_](https://en.wikipedia.org/wiki/Gnathostomata) (the infraphylum of jawed vertebrates) are not adequate replacements for _fish(1)_. What is the motivation for this?
+
+Is Soares perhaps suffering from the common misconception that words can only have a single definition? But it's actually not uncommon for words in natural languages to have more than one (related) meaning, which can be distinguished from context. (That's why dictionaries have multiple numbered definitions under the same word with the same etymology.)
+
+For example, _water_. The word "water" can be used to mean H₂O in any form (in which sense ice is a kind of water), or specifically liquid H₂O (in which sense ice is not a kind of water). If someone says "water" and you're not sure if they're using it in the ice-inclusive or the ice-exclusive sense, and ice happens to be relevant to the conversation you're having, then you might have to ask the speaker for clarification! Fortunately, this doesn't cause a whole lot of problems among people who are trying to communicate with each other and don't [have an incentive](https://www.lesswrong.com/posts/onwgTH6n8wxRSo2BJ/unnatural-categories-are-optimized-for-deception) to start a pointless [dispute over definitions](https://www.lesswrong.com/posts/7X2j8HAkWdmMoS8PE/disputing-definitions).
+
+But if someone were to declare that water should _only_ be used in the ice-exclusive sense, and that pedants who want to want to talk about _water_ in the ice-inclusive sense are engaging in "definitional gynmastics" and need to invent a new word for their thing, that would be pretty weird ... right?
+
+Frankly, I'm puzzled. Nate Soares, famous [autodidact extraordinaire](https://www.lesswrong.com/posts/w5F4w8tNZc6LcBKRP/on-learning-difficult-things) and Executive Director of the Machine Intelligence Research Institute, is no doubt more intelligent and knowledgable than a humble ordinary programmer like myself. He clearly shares my passion for the philosophy of language. So whatever arguments _I_ can discover, surely he would have already invented independently. So I _must_ be missing something.
+
+Could there, perhaps, be some _additional context_ to this conversation that Soares neglected to make explicit? That seems unlikely, however.
+
+Anyway, this concludes my blog post about why I think it makes sense to use the word _fish_ in the sense of "cold-blooded vertebrate animal that lives in water, moving with the help of fins and breathing with gills" in many contexts, albeit possibly not all contexts. Soares's work is very important and I'm sure he's very busy, but since he seems to be so passionate on this issue, I wonder if he could spare a few moments to engage with my arguments? If so, I eagerly await his reply.
diff --git a/content/2021/unnatural-categories-are-optimized-for-deception.md b/content/2021/unnatural-categories-are-optimized-for-deception.md
new file mode 100644
index 0000000..d69a288
--- /dev/null
+++ b/content/2021/unnatural-categories-are-optimized-for-deception.md
@@ -0,0 +1,366 @@
+Title: Unnatural Categories Are Optimized for Deception
+Date: 2021-01-08 12:54
+Status: published
+Category: philosophy
+Tags: rationality, honesty, philosophy of language
+Slug: unnatural-categories-are-optimized-for-deception
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/onwgTH6n8wxRSo2BJ/unnatural-categories-are-optimized-for-deception)
+
+**Followup to**: [Where to Draw the Boundaries?](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries)
+
+_There is an important difference between having a utility function defined over a statistical model's performance against specific real-world data (even if another mind with different values would be interested in different data), and having a utility function defined over features of the model itself._
+
+_Arbitrariness in the map doesn't correspond to arbitrariness in the territory. Whatever criterion your brain is using to decide which word you want,_ is _your non-arbitrary reason ..._
+
+So the one comes back to you and says:
+
+> That seems wrong—why wouldn't I care about the utility of having a particular model? I agree that categories derive much of their usefulness from "carving reality at the joints"—that's _one_ very important kind of consequence of choosing to draw category boundaries in a particular way. But other consequences might matter too, if we have some _moral_ reason to _value_ drawing our categories a particular way. I don't see why I shouldn't be willing to trade off one unit of categorizational nonawkwardness for $X$ units of morality, even if trading off a million units of categorizational nonawkwardness for the same $X$ units of morality would be bad.
+>
+> I once read about [an analogy between category boundaries and national borders](https://www.lesswrong.com/posts/aMHq4mA2PHSM2TMoH/the-categories-were-made-for-man-not-man-for-the-categories). Imagine a diplomat trying to come up with a proposal for a [two-state solution to the Israeli–Palestinian conflict](https://en.wikipedia.org/wiki/Two-state_solution). There's no such thing as the "correct" border between Israel and Palestine, but there are _consequences_ of choosing one border or another. For example, awarding territory to one side risks angering the other. For another, if the West Bank and Gaza Strip are to be part of Palestine, but Tel-Aviv and the southern city of Eilat are to be part of Israel, then topology forces you to decide which of Israel and Palestine gets to be continuous, and which will be split into two parts, because a "land bridge" between Gaza and the West Bank would separate Tel Aviv and Eilat, and _vice versa_. Since borders can't be "true" or "false", the diplomat's task is and _can only be_ to weigh these kinds of trade-offs.
+>
+> Analogously, I think of language, following Eliezer Yudkowsky's ["A Human's Guide to Words"](https://www.lesswrong.com/s/SGB7Y5WERh4skwtnb), as being a human-made project intended to help people understand each other. It draws on the structure of reality, but has many free variables, so that the structure of reality doesn't constrain it completely. This forces us to make decisions, and since these are not about factual states of the world—what the definition of a word _really_ is, in God's dictionary—we have nothing to make those decisions on except consequences.
+
+... okay, I think I see the problem. I see how one might have gotten that out of "A Human's Guide to Words"—_if you skipped all the parts with math_. I am now prepared to explain _exactly_ what's wrong here [in _more detail_](https://www.lesswrong.com/posts/2TPph4EGZ6trEbtku/explainers-shoot-high-aim-low) than [my previous attempt](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries): not just _that_ this position is not in harmony with the [hidden Bayesian structure](https://www.lesswrong.com/posts/QrhAeKBkm2WsdRYao/searching-for-bayes-structure) of language and cognition, but how the hidden Bayesian structure of language and cognition explains _why_ an intelligent system might find this particular mistake _tempting_ in the first place, and what breaks as a result.
+
+Category "boundaries" are a useful _visual metaphor_ for helping explain the cognitive function of categorization. If you have the visualization but you _don't_ have the math, you might think you have the freedom to "redraw" the category "boundaries". Simple, compact boundaries might _tend_ to be more useful, but more complicated boundaries aren't _false_ and therefore aren't forbidden if you have some non-epistemic reason to prefer them ... right?
+
+Only in the sense that _no_ hypothesis is "false"! Categories, words, correspond to hypotheses—probabilistic models that [make predictions](https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences). If I see a dolphin in the water, and I say, "Hey, there's a dolphin!", and you _understand_ me, that enables you to predict quite a lot about there being [this-and-such kind of aquatic mammal with fins, a tail, _&c._](https://en.wikipedia.org/wiki/Dolphin) in the water.
+
+This AI capability of "speech" is not only very powerful; it's also easy to understand the [cause-and-effect evidential entanglement](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence) [which explains _how it works_](https://www.lesswrong.com/posts/fKiTt55jEiTFK5prp/entangled-with-reality-the-shoelace-example)—at least at a very high level.
+
+Photons bounce off the dolphin and hit my eyes. I recognize the photons as forming an image that matches a concept that I associate with the word/symbol "dolphin" (implementation details omitted). I emit a "dolphin" signal composed of sound waves which hit your eardrum. By [a convention that culturally evolved due to our predecessors having a shared interest in communicating with each other, you map the "dolphin" signal](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution) to an internal concept that closely resembles the one I associate with that same signal. This works because we happen to live in a world where the distribution of creatures has [cluster-structure](https://www.lesswrong.com/posts/WBw8dDkAWohFjWQSk/the-cluster-structure-of-thingspace) whereby dolphins have lots of things in common with each other, such that it's possible to use observations about an entity to infer that it "is a dolphin", and then use the _dolphin_ concept to make good predictions about aspects of the entity that have not yet been observed; we owe our confidence that we've learned "the same" _dolphin_ model to the fact that dolphins actually exist.
+
+But the _dolphin_ concept/model/hypothesis is subject to the universal [_mathematical laws_](https://www.lesswrong.com/posts/CPP2uLcaywEokFKQG/toolbox-thinking-and-law-thinking) of reasoning under uncertainty. In particular, probability-mass flows _between_ hypotheses: as long as you [never assign a probability of _zero_](https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-are-not-probabilities) (which is a log-odds of negative infinity), nothing you believe can ever be _definitively_ [(infinitely)](https://www.lesswrong.com/posts/ooypcn7qFzsMcy53R/infinite-certainty) "falsified"—it "just" makes quantitatively worse predictions as _compared to_ other hypotheses.
+
+Because category "boundaries" are merely a visualization for a probabilistic model that makes predictions about the real world, you _can't_ "redraw the boundaries" associated with a communication signal without messing with the model that generates them, which means messing with your predictions about the real world.
+
+Might there be some non-epistemic reason for an agent to prefer a model that makes worse predictions? Sure! Correct maps are useful for [steering reality into configurations ranked higher in your preference ordering](https://www.lesswrong.com/posts/D7EcMhL26zFNbJ3ED/optimization)—but causing a _different_ agent to have _incorrect_ maps might make them _mis_-navigate reality in a way that benefits you! We call this [_deception_](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception).
+
+In a related phenomenon, a poorly-designed agent might get confused and end up manipulating its _own_ beliefs: optimizing its map to _inaccurately_ portray a high-value territory (rather than optimizing the territory to be high-value by using a map that reflects the territory), a kind of _self_-deception. We call this [_wireheading_](https://www.lesswrong.com/posts/aMXhaj6zZBgbTrfqA/a-definition-of-wireheading).
+
+The laws of probability and information theory allow us to calculate how information can be efficiently encoded and transmitted from one place to another. Given some distribution of random variables, and some specification of what information about those variables you want to transmit, some encodings—some ways of "drawing" category "boundaries"—quantitatively _perform better_ than others. Agents that _want to communicate with each other_ will tend to invent or discover conventions that efficiently encode the information they're trying to communicate. Agents that communicate in ways that systematically depart from efficient encodings are [better modeled as](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie) trying to deceive each other or wirehead themselves.
+
+-----
+
+Let's walk through a simple example. [Imagine that you have a peculiar job in a peculiar factory](https://www.lesswrong.com/posts/4FcxgdvdQP45D6Skg/disguised-queries): specifically, you're a machine-learning engineer tasked with automating away the jobs of humans who sort objects from a mysterious conveyor belt.
+
+Another engineer has already written a system that processes camera and sensor data about the objects into more convenient ["features"](https://en.wikipedia.org/wiki/Feature_(machine_learning)): color (measured on an eight-point blueness scale), shape (measured on an eight-point "eggness" scale), and vanadium content (a boolean Yes or No). Your task is to further process this information into a format suitable for giving commands to other systems—for example, the robot arm that will physically move the objects into appropriate bins.
+
+The feature data consists of the blueness–eggness–vanadium-content joint distribution given by this 128-entry table:
+
+![blueness–eggness–vanadium joint distribution](https://i.imgur.com/zR83zOq.png)
+
+This seems like ... not the most useful representation? The data is all there, so _in principle_, you could code whatever you needed to do based off the full table, but it seems like it would be an unmaintainable mess: you'd sooner _resign_ than write a 128-case [switch statement](https://en.wikipedia.org/wiki/Switch_statement). Furthermore, when the system is deployed, you hope to typically be able to give the binning robot messages based on _only_ the color and shape observations, because the Sorting Scanner that the vanadium readings come from is expensive to run. You _could_ just do a Bayesian update on the entire joint distribution, of course, but it seems like it should be possible to be more efficient by exploiting regularities in the data, not entirely unlike how your colleague's system has _already_ made your job much simpler by giving you blueness and eggness feature scores rather than raw camera data. Eyeballing the table, you notice it seems to have a lot of redundancy: most of the probability-mass is concentrated in two regions where the blueness and eggness scores are either both high or both low—and vanadium is _only_ found when both blueness and eggness are high.
+
+O tragedy O the stars! _If only_ there were _some more convenient and flexible way_ to represent this knowledge—some kind of deep structural insight to rescue you from this cruel predicament!
+
+... alright, dear reader—I shouldn't patronize. [You already know how this story ends.](https://www.lesswrong.com/posts/gDWvLicHhcMfGmwaK/conditional-independence-and-naive-bayes) The distribution factorizes!
+
+$$\sum_{\mathrm{category}} P(\mathrm{category}) \cdot P(\mathrm{blueness}|\mathrm{category}) \cdot P(\mathrm{eggness}|\mathrm{category}) \cdot P(\mathrm{vanadium}|\mathrm{category})$$
+
+(The distribution in this made-up toy example factorizes _exactly_, but in a messy real-world application, you might have a spectrum of approximate models to choose from.)
+
+We can simplify our representation of our observations by using a [naïve Bayes model](https://en.wikipedia.org/wiki/Naive_Bayes_classifier), a "star-shaped" [Bayesian network](https://www.lesswrong.com/posts/hzuSDMx7pd2uxFc5w/causal-diagrams-and-causal-models) where a central "category" node is posited to underlie all of our observations: we believe that each object either "is a blegg" (and therefore contains vanadium and has high blueness and eggness scores) with probability 0.48, "is a rube" (and therefore has no vanadium and low blueness and eggness scores) with probability 0.48, or belongs to a catch-all "other"/error class with probability 0.04. (Maybe the camera is buggy sometimes, or maybe there are some other random objects mixed in with the rubes and bleggs?)
+
+![factorized object distribution](https://i.imgur.com/zIaDccJ.png)
+
+The full joint distribution had 127 degrees of freedom (a table of $8 \cdot 8 \cdot 2 = 128$ separate probabilities, constrained to add up to 1), whereas the naïve-Bayes representation only needs 57 parameters ($3 \cdot 1$ prior probabilities for the categories, plus $3 \cdot 8 = 24$, $3 \cdot 8 = 24$, and $3 \cdot 2 = 6$-entry _conditional_ probability tables for each of the features). The advantage would be much larger for more complicated problems: the joint distribution table grows exponentially with more features, quickly becoming infeasible to _store and represent_, let alone _learn_.
+
+It must be stressed that our "categories" here are a _specific mathematical model_ that makes _specific_ (probabilistic) predictions. Suppose we see a black-and-white photo of an egg-shaped object: specifically, one with an eggness score of 7. Given that observation of $\mathrm{eggness} = 7$, we can update our probabilities of category-membership.
+
+$$P(\mathrm{category} = c | \mathrm{eggness} = 7) = \frac{P(\mathrm{eggness} = 7|\mathrm{\mathrm{category} = c})P(\mathrm{category} = c)}{\sum_{d \in \{\mathrm{blegg}, \mathrm{rube}, \mathrm{??} \} } P(\mathrm{eggness} = 7| \mathrm{category}=d)P(\mathrm{category} = d)}$$
+
+We think the egg-shaped object is almost certainly a blegg (specifically, with probability 0.96), even if the black-and-white photo doesn't directly tell us how blue it is, _because_
+
+$$P(\mathrm{category} = \mathrm{blegg} | \mathrm{eggness} = 7) = \frac{\frac{1}{4} \cdot \frac{12}{25}}{\frac{1}{4} \cdot \frac{12}{25} + 0 \cdot \frac{12}{25} + \frac{1}{8} \cdot \frac{1}{25}} = \frac{24}{25} = 0.96$$
+
+We can then use our updated beliefs about category membership (0.96 blegg/0 rube/0.04 unknown, as contrasted to the 0.48/0.48/0.04 prior) to get our updated posterior distribution on the 0–7 blueness score (0.005/0.005/0.005/0.005/0.005/0.245/0.485/0.245—left as an exercise for the reader).
+
+------
+
+In addition to categories facilitating efficient probabilistic inference _within_ the system that you're currently programming, _labels_ for categories turn out to be useful for _communicating_ with other systems. The robot arm in the Sorting room puts bleggs in a blegg bin, which gets taken to a room elsewhere in the factory where there's sophisticated vanadium-ore-processing machinery that has to handle both bleggs and gretrahedrons.
+
+But suppose the binning arm doesn't need to _know_ about the blueness and eggness scores: it can close its claws around rubes and bleggs alike, and you only need to program it to pick up an object from a certain spot on the conveyor belt and place it into the correct bin. However, the vanadium-ore-processing machine does need to do further information processing before it can operate on an object—perhaps it needs to vary its drill speed in proportion to the density of a particular blegg's flexible outer material (which it can estimate based on how brightly the blegg glows in the dark), but it uses a different drilling pattern for gretrahedrons.
+
+If you need to send commands to both the binning arm and the ore-processing machine, it's a more efficient communication protocol to just be able to send the 28-byte [JSON](https://www.json.org/json-en.html) payload `{"object_category": "BLEGG"}` and let the other machines do their work using their _own_ models of bleggs, rather than having to send over the raw camera data plus the binary code of the Bayesian network and feature extractors that you initially used to _identify_ bleggs. [Intelligence is prediction is compression](https://www.lesswrong.com/posts/ex63DPisEjomutkCw/msg-len): our ability to find an encoding that [compresses the length of the message](https://www.lesswrong.com/posts/mB95aqTSJLNR9YyjH/message-length) needed to convey information about the objects is [fundamental to our having _learned_ something](https://www.lesswrong.com/posts/hAvGi9YAPZAnnjZNY/prediction-compression-transcript-1) about the distribution of objects.
+
+The `{"object_category": "BLEGG"}` message is a useful shorthand for "linking up" the models between different machines. Different machines might not use the _same_ model: the classifier system uses blueness and eggness scores to _identify_ bleggs, but the ore-processing machine, having been _told_ that an object is a blegg, can take its approximate blueness and eggness for granted and only needs to reason about its luminescence and vanadium content.
+
+But this trick of using a signal to correlate the models between different machines only works _because_ and _insofar as_ both models are pointing to the same cluster-structure in reality. If the model in the classifier system doesn't meaningfully _match_ the model in the ore-processing system—if the classifier code sends the `{"object_category": "BLEGG"}` message given a object with blueness score between 5 and 7, but the ore-processor, upon receiving the `{"object_category": "BLEGG"}` message, positions its drills in the expectation of processing an object with an eggness score between 0 and 2—then the factory doesn't work.
+
+-----
+
+As a human learning math, it's helpful to examine [multiple representations of the same mathematical object](https://en.wikipedia.org/wiki/Multiple_representations_(mathematics_education)). We've already seen our blueness–eggness–vanadium model represented as a table, and factorized into a graphical model. We've done also some algebraic calculations with it. But we can also visualize it: the set of camera observations that the model classifies as a blegg with probability $\ge 0.96$ can be thought of a area with a boundary in two-dimensional blueness–eggness space:
+
+![](https://i.imgur.com/lcapjbb.png)
+
+("With probability $\ge 0.96$" because our catch-all "other"/error category can also generate examples with high blueness and eggness scores; we can't say things like "Everything inside the boundary in the diagram is a blegg" when we're talking about a formal model where some of the categories generate overlapping observations in whatever subspace the diagram is depicting.)
+
+If you were trying to _teach_ someone about the hidden Bayesian structure of language and cognition, but thought your audience was too stupid or lazy to understand the actual math, you might be tempted to skip the part about factorizing a joint distribution into a star-shaped Bayesian network and just talk about "drawing" "boundaries" in configuration space for human convenience, perhaps with a hokey metaphor about national borders. Then the audience might walk away with the idea that there's no reason not to replace the old _blegg_ concept and its boring compact boundary, with a new _blegg\*_ concept that has an exciting squiggly border.
+
+Alaska [isn't even _contiguous_ with](https://en.wikipedia.org/wiki/Contiguous_United_States) the rest of the United States. If _that's_ okay, why can't the borders of bleggness be a little squiggly?
+
+![](https://i.imgur.com/IBKatUG.png)
+
+Because the "national borders" metaphor is [just a metaphor](https://www.lesswrong.com/posts/C4EjbrvG3PvZzizZb/failure-by-analogy). It _immediately_ breaks down as soon as you try to do any calculations.
+
+When we say that [the United States purchased Alaska from the Russian Empire](https://en.wikipedia.org/wiki/Alaska_Purchase), that _means_ that this-and-such physical area on the Earth's surface went from being the territory of the Russian government, to being territory of the United States government, where land being the "territory of" a "government" is a complicated idea that has something to do [Schelling points over who gives orders to policemen and soldiers in that area](https://www.lesswrong.com/posts/YMtZRGLbvdD4BGaqN/generalized-efficient-markets-in-political-power#Governance_as_Schelling_Point).
+
+When you reprogram your machine-learning system to send an `{"object_category": "BLEGG"}` message when it sees an object with an eggness score of 2 and a blueness score of 1, then your vanadium-ore-processing machine wears down its drill bits trying to process a rube.
+
+_Other than_ the fact that _some aspects_ of both of these situations can be usefully _visualized_ as changes to a two-dimensional diagram depicting an area with a boundary, what do these situations have to do with each other? They don't. Countries aren't Bayesian networks. They just aren't. When we depict a country on a map, we're _not talking_ about a cognitive system that can use observations of latitude to estimate probabilities of country-membership and then use that distribution on country-membership to get an updated probability distribution on longitude. (I mean, given a world map, you _could_ program such a thing, but it seems kind of useless—it's not clear why anyone would _want_ that particular program.) Why would you expect to understand an AI-theory concept by telling a story about national borders?
+
+------
+
+So, that's what's wrong with the national-borders metaphor. But we haven't yet really explained the problem with "unnatural" categories—those that you would _visualize as_ a squiggly, "gerrymandered" boundary. The squiggly _blegg\*_ boundary doesn't have the nice property of corresponding to the category labels in our nice factorized naïve Bayes model, but it still contains information. You can still do a Bayesian update on being told that an object lies within a squiggly boundary in configuration space. If that update eliminates half of your probability-mass, that's one information-theoretic bit, no matter how the category is shaped in Thingspace.
+
+If you only care about how much probability you assign to the _exact_ answer, then a bit is a bit. But if an approximate answer is approximately as good—if your answerspace has a metric on it, so that "approximate" can mean something—then some bits can be more valuable than others.
+
+Suppose some random variable $X$ is uniformly distributed on the set $\{1, 2, 3, 4, 5, 6, 7, 8\}$. You have the option of being told either whether an observation $x$ sampled from $X$ is even or odd, or whether $x$ is greater or less than 4.5. Either way, you eliminate half of your hypotheses: the [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)) of your probability distribution goes from $\log_2 8 = 3$ to $\log_2 4 = 2$. Either way, you've learned 1 bit.
+
+Still, if you have to make a decision that depends on "how big" $x$ is, it seems like the "1–4 or 5–8" category system is going to be more useful than the "even/odd" category system, even though they both provide the same amount of information about the _exact_ answer. If you learn that $x \in \{1, 2, 3, 4\}$, then you know that $x$ is "small", but if you learn that $x$ is odd, you haven't learned much about how big it is: it could be 1, but it could just as well be 7.
+
+To formalize this, let's measure how "good" a category is using the [expected squared error](https://en.wikipedia.org/wiki/Mean_squared_error). "Error" is how much a prediction is wrong by: if you guessed $x$ was 2, but it was actually 5, your error would be $5 - 2 = 3$, and your squared error would be the square of that, $3^2 = 9$. The expected squared error of a probability distribution is, on average, the square of how much your guess about a sample from that distribution will be wrong. [(The squared error has nicer mathematical properties than the absolute error.)](https://www.benkuhn.net/squared/)
+
+For our example of $x$ sampled from $X$ uniformly distributed on $\{1, 2, 3, 4, 5, 6, 7, 8\}$, your best-guess estimate $\hat{x}$ of $x$ is going to be the expected value
+
+$$\sum_{x\in\{1...8\}}P(X=x)\cdot x=\frac{1+2+3+4+5+6+7+8}{8}=4.5$$
+
+And the initial expected squared error is
+
+$$E[(x-\hat{x})^{2}]=\sum_{x\in\{1...8\}}P(X=x)\cdot(x-\hat{x})^{2} =\frac{(1-4.5)^{2}+(2-4.5)^{2}+...+(8-4.5)^{2}}{8}=5.25$$
+
+Suppose you then learn whether $x$ is even or odd.
+
+With probability 0.5, you learn that $x$ is even. In that case, your new estimate $\hat{x}$ taking that into account would be
+
+$$\sum_{x\in\{2,4,6,8\}}P(X=x)\cdot x=\frac{2+4+6+8}{4}=\frac{20}{4}=5$$
+
+and your new expected squared error (in the "even" possible world) would be
+
+$$E[(x-\hat{x})^{2}]=\sum_{x\in\{2,4,6,8\}}P(X=x)\cdot(x-\hat{x})^{2}=\frac{(2-5)^{2}+(4-5)^{2}+(6-5)^{2}+(8-5)^{2}}{4}$$
+
+$$= \frac{9+1+1+9}{4}=\frac{20}{4}=5$$
+
+With probability 0.5, you learn that $x$ is odd. Similar calculations (left as an exercise) also give a new expected squared error of 5 in the "odd" possible world. Averaging over both cases (trivially, $0.5 \cdot 5 + 0.5 \cdot 5 = 5$), learning whether $x$ is even or odd only brought our expected squared error down from 5.25 to 5, barely changing at all.
+
+In contrast, if you learn whether $x$ is 1–4 or 5–8, your expected squared error plummets to 1.25. (Exercise.) By being compact, the "1–4 or 5–8" category system is much more useful for getting _close_ to the right answer than the "even/odd" category system.
+
+The same goes for natural categories _versus_ squiggly category "boundaries" in configuration space; we just need to supply some [metric](https://en.wikipedia.org/wiki/Metric_(mathematics)) to define what "close" means.
+
+For our blueness–eggness–vanadium distribution, suppose we use the [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance) on blueness-score ✕ eggness-score ✕ 1-if-vanadium-present-else-0. (So, for example, the "distance" between the typical blegg and the typical rube is $\sqrt{(6 - 1)^2 + (6 - 1)^2 + (1 - 0)^2} = \sqrt{25 + 25 + 1} = \sqrt{51} \approx 7.14$ under this metric.)
+
+Then our expected squared error before being told anything about an object is about 13.63. On being told whether an object is a blegg, rube, or other (according to the categories in our nice factorized naïve Bayes model), our expected squared error plummets to 1.38.
+
+But suppose that, instead of our nice factorized naïve Bayes model, we use a category system based on drawing squiggly "boundaries" in configuration space: everything inside the blegg\* boundary in the diagram is a blegg\*, everything within the rube\* boundary in a rube\*, and anything outside belongs to a catch-all "other\*" category.
+
+![](https://i.imgur.com/KuBZZkO.png)
+
+On learning whether an object is a blegg\*, rube\*, or other\*, our expected squared error only goes down to about 4.12.[^script]
+
+[^script]: [The source code of the Python script used for these calculations is available.](https://gist.github.com/zackmdavis/46c3a1ab346c9f548b7e2fce2e955a78)
+
+In this sense, the gerrymandered blegg\* concept is _quantitatively less informative_ than the original, compact blegg concept. The _metric_ we assigned to blueness–eggness–vanadium space was our choice, and could depend on our values: for example, if we simply _don't care_ about predicting how blue an object is, we could disregard the blueness score and only define a concept on the eggness–vanadium subspace (in which case our initial expected squared error is about 6.94, plummets to 0.69 given knowledge of blegg/rube/other category-membership, but only goes down to about 1.81 given knowledge of the gerrymandered blegg\*/rube\*/other\* category). Or if we don't care about predicting blueness _very much_, we could calculate our error score with respect to a metric that gave blueness very little weight. (Exercise.)
+
+But _given_ a metric on the variables that you care about predicting and using to inform predictions, which categories are cognitively useful depends on the the distribution of data in the world. You can't define a word any way you want.
+
+------
+
+The dependence on a choice of metric on configuration space—and really, a choice of the space—gives a _sense_ in which optimal categories are [value-laden](https://arbital.greaterwrong.com/p/value_laden), but it's a specific kind of _lawful_ dependence between your values and the distribution of data in the world, _not_ an atomic preference for using a particular encoding for its own sake.
+
+The cognitive function of categorization is to group similar things together so that we can make similar decisions about them. A function measuring the extent to which things are "similar" has to take the things as input, but the extent to which things are _decision-relevantly_ similar also depends on what you're trying to accomplish with your decisions, and that can be algorithmically complex. It might not be just a matter of only looking at some decision-relevant subspace of a natural, "obvious" configuration space that's available to [all possible minds](https://www.lesswrong.com/posts/tnWRXkcDi5Tw9rzXw/the-design-space-of-minds-in-general) (like not caring what color your toothbrush handle is—um, if we pretend that all possible minds had human-like color vision); the dimensions of the space you do your [similarity-clustering](https://www.lesswrong.com/posts/jMTbQj9XB5ah2maup/similarity-clusters) in might themselves be complicated features (in the sense of machine learning) of which agents with different values would have no reason to [logically pinpoint](https://www.lesswrong.com/posts/3FoMuCLqZggTxoC3S/logical-pinpointing) that particular criterion [by which things may be judged](https://www.lesswrong.com/posts/zqwWicCLNBSA5Ssmn/by-which-it-may-be-judged). How you should define words _depends on_ what you want, but that's _not_ the same as defining words any way you want.
+
+For example, [_poison_ isn't a natural category to a generic mind studying chemistry](https://www.lesswrong.com/posts/XeHYXXTGRuDrhk5XL/unnatural-categories): we group cyanide and hemlock together as _poison_ because we value human health, and so we want to have a category for scary chemicals that disrupt human metabolism, causing death or serious illness. But this determination depends on the intricate details of human biochemistry. (The [theobromine in chocolate](https://en.wikipedia.org/wiki/Theobromine_poisoning) is okay for humans at typical doses, but potentially fatal to dogs, which are actually pretty close to us in animalspace.) The compact category "boundary" that minimizes predictive error on human-healthspace, corresponds to a squiggly "boundary" in the chemicalspace you would be looking at if you've never seen a human and just want to make predictions about the chemicals themselves.
+
+Or [tiny molecular smileyfaces and real human smiles might be grouped together](https://www.lesswrong.com/posts/PoDAyQMWEXBBBEJ5P/magical-categories) as similar as far as an image-classifier's [curve detector](https://distill.pub/2020/circuits/curve-detectors/) is concerned, even if they're not similar as far as the [abstracted idealized dynamic](https://www.lesswrong.com/posts/9KacKm5yBv27rxWnJ/abstracted-idealized-dynamics) of human morality is concerned.
+
+The technical sense in which optimal categories can be value-laden doesn't alter the basic morals of our basic Bayesian philosophy of language. Your values can give you a particular configuration space and a metric on the space, but _given_ that, sane agents want to "carve it at the joints" in order to get a communication system that minimizes predictive error. If you're trying to find an efficient encoding of your observations, there's no reason to _want_ squiggly, gerrymandered categories in the decision-relevant space.
+
+------
+
+The one replies:
+
+> You're still not addressing my crux! I don't doubt what you say about minimizing prediction error with respect to some squared metric thingy. But what if that's not what I care about? _My_ utility function assigns high value to using the squiggly _blegg\*_ category boundary—such that the utility of using my preferred category outweighs the disutility of making less accurate predictions. You _can_ define a word any way you want—if you're willing to pay the costs.
+
+So, what, you just intrinsically assign high utility to using the same communication signal to encode eggness-2/blueness-1 observations as eggness-6/blueness-6 observations, given the joint distribution specified in my story problem about sorting objects in a factory? Really?
+
+"... yes!"
+
+Okay, but where would that kind of exotic utility function come from? How would it arise naturally in an intelligent system?
+
+There's a _trivial_ sense in which you can interpret any action taken by an agent as being taken because the agent _values taking that action_. This theory [is compatible with all possible behaviors and therefore explains nothing](https://www.lesswrong.com/posts/jiBFC7DcCrZjGmZnJ/conservation-of-expected-evidence).
+
+The value of [decision-theoretic utility functions](https://www.lesswrong.com/posts/DQ4pyHoAKpYutXwSr/underappreciated-points-about-utility-functions-of-both) isn't that "Because utility!" serves as [an all-purpose excuse for any possible behavior](https://www.lesswrong.com/posts/i6fKszWY6gLZSX2Ey/fake-optimization-criteria). It's that [simple coherence desiderata _imply_ that an agent's behavior should be _describable as_ maximizing expected utility for some utility function](https://www.lesswrong.com/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities)—with corresponding _constraints_ on the shape of that behavior.
+
+Situations like [the Allais paradox](https://www.lesswrong.com/posts/zJZvoiwydJ5zvzTHK/the-allais-paradox) illustrate what these constraints look like. Consider an AI faced with playing the following game. There's a switch that can be turned On or Off, that starts out on in the Off position. At midnight, a coin is flipped. If the coin comes up Tails, the game ends. If the coin comes up Heads, then at a quarter past midnight, if the switch is Off, then the AI gets paid $100, and if the switch is On, a six-sided die is rolled, and the AI gets paid $110 if the die doesn't come up 6.
+
+Suppose that, before midnight, the AI is willing to pay a dollar to flip the switch On (as if it thought that winning $110 with a probability of 5/12 is better than winning $100 with a probability of 1/2). Suppose the coin comes up Heads, and the AI is then willing to pay another dollar to flip the switch Off again (as if it thought that $100 with certainty is better than $110 with probability 5/6). Then the AI is two dollars poorer in exchange for the switch being in the same position it started in.
+
+These gambling preferences violate [the independence axiom](https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem#The_axioms) of the [von Neumann–Morgenstern utility theorem](https://www.lesswrong.com/posts/F46jPraqp258q67nE/why-you-must-maximize-expected-utility). You _can't_ have a utility function $U$ for which 
+
+$$\frac{1}{2} \cdot U(\$100) \lt \frac{5}{12} \cdot U(\$ 110)$$
+
+and
+
+$$U(\$100) \gt \frac{5}{6} \cdot U(\$110)$$
+
+because the sides of the second inequality are just those of the first multiplied by two, and multiplying by two should preserve the direction of inequality.
+
+Having shown this, can we say that an AI with such behavior is "irrational"? But what does that even mean? If, for some reason, you specifically programmed the AI to prefer options it considers "certain", or to want switches to be "On" before midnight but "Off" after midnight, then it would be functioning as designed.
+
+What we _can_ say about such an AI, is that it doesn't have a utility function [_in terms of money_](https://www.lesswrong.com/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities?commentId=GyE8wvZuWcuiCaySb), and is therefore not coherently optimizing for acquiring money. Recall that we say that a system is an optimizer if it systematically [steers the future](https://www.lesswrong.com/posts/HktFCy6dgsqJ9WPpX/belief-in-intelligence) into configurations that rank higher [with respect to some preference ordering](https://www.lesswrong.com/posts/CW6HDvodPpNe38Cry/aiming-at-the-target). This helps us make predictions about what _effects_ the system has, without having to model the details of _how_ it brings those effects about. A well-designed agent that was optimizing for acquiring money would be expected to obey the independence axiom.
+
+If the AI playing this game isn't coherently optimizing for acquiring money, what _is_ it optimizing for? To tell, we'd need to observe its behavior in different environments and see how it [responds to perturbations](https://www.lesswrong.com/posts/znfkdCoHMANwqc2WE/the-ground-of-optimization-1). If it is trying to acquire money but is just _biased_ to prefer certainty (in violation of the von Neumann–Morgenstern axioms), then we'd expect it to make choices that result in money but continue to exhibit Allais-like glitches around gambles involving probabilities close to 1. If it just likes switches to be off after midnight, then we'd expect it to turn switches off at that time even if there's no gambling game going on.
+
+This methodology for attributing goals to an agent—consider it to be "optimizing for" outcomes that it [systematically achieves across a variety of environments](https://www.lesswrong.com/posts/yLeEPFnnB9wE7KLx2/efficient-cross-domain-optimization)—applies to the behavior of sending communication signals, just as it does to the behavior of flipping switches.
+
+Back to the factory. Our classifier system sends a `{"object_category": "BLEGG"}` message when it gets feature data corresponding to the compact _blegg_ concept. This behavior is optimized for sending messages that allow other systems to minimize the expected squared error of their predictions of objects with respect to our standard metric on blueness–eggness–vanadium space. We _don't_ intrinsically "assign utility" to using that particular category system; the category is the _solution_ to an optimization problem about how to efficiently get blueness–eggness–vanadium information from one place to another.
+
+A system that sends a `{"object_category": "BLEGG"}` message when it gets camera data corresponding to the gerrymandered _blegg\*_ concept would be optimized for ... what? If you don't intrinsically assign utility to using that particular category system, then _why_ would you program the system that way? What could possibly be the problem for which the gerrymandered category is an optimized solution?
+
+Well. Suppose that, besides your dayjob as a machine-learning engineer, you _also_ happen to own a side interest in the firm that supplies bleggs and rubes to this very factory. And suppose that vanadium fetches higher market prices than palladium, such that the factory is to pay the supplier $2 per blegg but only $1 per rube—and that the accounts-payable records are to be compiled based on how much the classifier you're currently programming sends `{"object_category": "BLEGG"}` and `{"object_category": "RUBE"}` messages, _not_ how much metal actually gets harvested.
+
+You can't help but notice that you stand to make more money if the system you're programming sends `BLEGG` messages more often. You can't just make it send `BLEGG` messages all the time—someone would notice and you'd get fired. But the ore-processing room can cope with a _few_ suboptimally-sorted objects. Surely it's no big deal if you just ... adjusted the category boundary of `BLEGG`-ness a bit?
+
+We saw earlier that the _blegg_ concept does better than the _blegg\*_ concept with respect to mean squared error (given a metric on the feature space).
+
+That's not the only possible scoring function with which one could formalize how "good" a category system is. Suppose that instead we score our category system by which one best minimizes the expected squared error _minus_ supplier revenue in cents. With respect to this criterion, accurate predictions are still good, but supplier revenue is _also_ good.
+
+Learning whether an object is a blegg, rube, or other (according to the "natural" categories in our naïve Bayes model) yields a squared-error-minus-revenue score of about −142.62. (Don't ask me what the units are on this.) But learning whether an object is a blegg\*, rube\*, or other\* yields a squared-error-minus-revenue of −151.57, which is lower (which is better, because we formulated this as a minimization problem). So with respect to _that_ scoring function, the _blegg\*_ category "boundary" is preferable.
+
+-----
+
+The one says:
+
+> But now it sounds like you're agreeing with me! The compact _blegg_ category serves the factory owner's goals better, which you formalized in terms of minimizing average squared error. The squiggly _blegg\*_ boundary makes the factory perform less well, but it serves the moonlighting engineer's goals better, which you formalized in terms of minimizing squared error minus supplier revenue. There's no rule of rationality against the engineer programming the system using the _blegg\*_ category boundary if it suits their goals better.
+
+Only in the sense that there's no rule of rationality against _lying!_ Suppose I'm selling you some number of gold and silver bars, but you can't examine the metal yourself until later; you can only hope that the receipt I give you is accurate. Consider the following two scenarios.
+
+In the first scenario, I _lie_: the receipt says I delivered 60 gold bars and 20 silver bars, but I actually delivered 40 gold bars and 40 silver bars. You live in a low-trust world where lying is very common and contract enforcement isn't really a thing: a third of the time an object is claimed to be gold, it turns out to be silver. So when you discover the fraud, you feel disappointed but not surprised: you would have _preferred_ to get what you paid for, but you can't say you _anticipated_ it.
+
+In the second scenario, I tell the truth—with respect to a category system that suits my goals. The receipt says I delivered 60 gold bars and 20 silver bars—and I did. It's just that what _I_ prefer to call "gold bars", you prefer to call "gold bars, _or_ silver bars with odd [serial numbers](https://en.wikipedia.org/wiki/Gold_bar#Security_features)", and what I call "silver bars", you call "silver bars with even serial numbers". You know this, so when you examine the actual contents of the delivery, you feel disappointed but not surprised: you would have _preferred_ to transact under your definitions of 'gold' and 'silver', but you can't say you _anticipated_ it.
+
+We might question whether these are two different scenarios, or two descriptions of the _same_ scenario: the same physical receipt, the same physical metal, _the same buyer anticipations about the metal conditional on observing the receipt_. If [we just pay attention to the evidential entanglements](https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-line) instead of being [confused by words](https://www.lesswrong.com/posts/KbjWLGJogCZY4HNsr/words-and-implications), then [there's no functional difference between](https://www.lesswrong.com/posts/YptSN8riyXJjJ8Qp8/maybe-lying-can-t-exist) saying "I reserve the right to lie _p_% of the time about whether something belongs to category _C_", and adopting a new, less-accurate category system that misclassifies _p_% of instances with respect to the old system.
+
+Minimizing the squared-error score is _about_ map–territory correspondence: ways of communicating that help the factory machines make better predictions about the objects, get a higher score.
+
+Minimizing the squared-error-minus-supplier-revenue score is a _compromise_ between map–territory correspondence and saying whatever makes the supplier the most money.
+
+The _degree_ of compromise is quantitative: there's a continuum of possible scoring functions between "minimize expected squared error, only" (for which the naïve-Bayes categorizer is a good solution), and "maximize supplier revenue, only" (for which "always say `BLEGG`" is the optimal solution). If always saying whatever profits you and not revealing _any_ information about the territory is deception pure and simple, then the intermediate points on a continuum with that can be thought of as partially deceptive.
+
+Depending on your goals, deception can be rational! If you _don't care_ about other agents having accurate models and just want to [intervene on them to make them believe](https://www.lesswrong.com/posts/Zvu6ZP47dMLHXMiG3/optimized-propaganda-with-bayesian-networks-comment-on) whatever makes them behave in a way that benefits you—or [whatever makes them happy](https://www.lesswrong.com/posts/synsRtBKDeAFuo7e3/not-for-the-sake-of-happiness-alone)—then you can do that! There's [no God to stop you](https://www.lesswrong.com/posts/sYgv4eYH82JEsTD34/beyond-the-reach-of-god). But in order to help you _decide_ whether deceiving people is the right thing to do, it helps to _notice_ that what you're doing is deceiving people.
+
+-----
+
+It helps to notice what you're doing—_if_ you're trying to be an agent that coherently steers the future in some direction. But who does that, really? Maybe you just want to _feel good!_ And not even coherently steer the universe into configurations where you feel good, either!
+
+Rational agents should want to have true beliefs: the map that reflects the territory, is the map that is _useful_ for navigating the territory. But you don't—can't—have unmediated access to the world; you can only _infer_ what the world is like from sensory data, and effectively [live in _your model of_ the world](https://www.lesswrong.com/posts/S8ysxzgraSeuBXnpk/rationality-quotes-july-2009?commentId=aNcj3BjsF2eXaDZHo). Given the tricky indirection involved, it's not surprising that [poorly-designed](https://www.lesswrong.com/posts/jAToJHtg39AMTAuJo/evolutions-are-stupid-but-work-anyway) agents like humans sometimes get confused and "wirehead" themselves: if you don't notice the difference, it's tempting to fabricate a fake map that _falsely_ portrays the territory as being good, instead of making a map that reflects the territory (which you can use to figure out how to improve the territory).
+
+Similarly, if you don't notice the difference, it's tempting to choose language that makes the world _sound_ good, than to have your language accurately describe the world (which description you can use to figure out how to make the world better).
+
+Suppose I want people to think I'm funny. _Funny_ is a value-laden concept in the specific lawful sense described earlier: non-human agents would have no motive to evaluate the particular [fixed computation](https://www.lesswrong.com/posts/FnJPa8E9ZG5xiLLp5/morality-as-fixed-computation) of humor. It's also a [fuzzy concept](https://www.lesswrong.com/posts/8gLEnEwm2g257vqyx/fuzzy-boundaries-real-concepts): we don't have a simple test to precisely measure in standard units exactly how funny a joke is, but there's _enough_ regularity in how people use the word "funny" for the word to be a useful communication signal. It's _also_ a [two-place concept](https://www.lesswrong.com/posts/eDpPnT7wdBwWPGvo5/2-place-and-1-place-words): people have different senses of humor, so that what I consider funny isn't exactly the same as what you consider funny.
+
+Given all these complications, one could imagine being tempted to think that humor is "subjective", and that therefore I can define it any way I want, and that therefore, if I feel sad about not being "funny", I can fix that by _changing my definition of the word "funny"_ such that it includes my jokes. Because definitions can't be "false", right!? There's no rule of rationality prohibiting this boundary-redrawing project—and since I want so desperately to be "funny", there's every rule of human decency in favor of it, right?!
+
+So, this obviously doesn't work. (Okay, it "works" if you deliberately choose to define the word "work" such that it works, but it doesn't _actually_ work.) [Yes requires the possibility of no](https://www.lesswrong.com/posts/G5TwJ9BGxcgh5DsmQ/yes-requires-the-possibility-of-no): redefining _X_ to make "Is it _X_?" come out true no matter what, [loses the purpose](https://www.lesswrong.com/posts/sP2Hg6uPwpfp3jZJN/lost-purposes) of asking the question in the first place. The proposal to redefine the word "funny" came with the purported justification that words don't have intrinsic meanings, so it can't be "wrong" to redefine it. But precisely _because_ words don't have intrinsic meanings, there's no reason to _want_ to redefine an _existing_ word, _except_ to piggyback off the meaning people are _already_ using that signal for.
+
+(Note that this, in itself, isn't necessarily deceptive. Sometimes, [coining new senses of a word that piggyback off an existing meaning can be a powerful tool for extending our vocabulary to cover new phenomena that we don't already have words for](https://www.lesswrong.com/posts/wR4PaDp2Knu5coeXx/metaphorical-extensions-and-conceptual-figure-ground)—as long as we're careful to [specify which meaning is intended](https://www.lesswrong.com/posts/shoMpaoZypfkXv84Y/variable-question-fallacies) when it's not clear from context.)
+
+It's not plausible to suppose that I want to be "funny" _because_ I like five-letter words that start with the letter _f_; I want to be funny _because_ of what that communication signal is already understood to refer to in [common usage](https://www.lesswrong.com/posts/9ZooAqfh2TC9SBDvq/the-argument-from-common-usage). The redefinition might (or might not) succeed at making me feel better about myself, but if it does, it only works _by means of_ confusing me: using [strategic equivocation](https://slatestarcodex.com/2014/11/03/all-in-all-another-brick-in-the-motte/) to arbitrage the hedonic gap between my new definition, and the old definition (which I still mentally associate with the word).
+
+If it _does_ succeed at making me feel better about myself, is the redefinition "rational"? Happiness is good, right? [Should not rationalists win?](https://www.lesswrong.com/posts/6ddcsdA2c2XpNpE5x/newcomb-s-problem-and-regret-of-rationality)
+
+I do not frame an answer: that would depend on how you draw the category boundaries of "rational", which is [not an interesting question](https://www.lesswrong.com/posts/7X2j8HAkWdmMoS8PE/disputing-definitions). (As it is written of a virtue which is nameless: if you speak overmuch of the Way, you will not attain it.)
+
+What I _can_ say, however, is that redefining the concept of humor is not a [procedure](https://www.lesswrong.com/posts/HcCpvYLoSFP4iAqSz/rationality-appreciating-cognitive-algorithms) that uses a map that reflects the territory to systematically achieve goals across a wide range of environments. If there's anything I can _do_ to _become_ funnier (like practicing telling jokes in a mirror, or studying great comedians to imitate their timing and delivery), I would seem less likely to notice and execute on such a plan after [having sabotaged](https://www.lesswrong.com/posts/Hs3ymqypvhgFMkgLb/doublethink-choosing-to-be-biased) the _concept_ I would need to notice the problem in the first place.
+
+-----
+
+The map is not the territory ... but for [real agents embedded in the physical universe](https://www.lesswrong.com/posts/i3BTagvt3HbPMx6PN/embedded-agency-full-text-version), the map is _part_ of the territory. This presents some complications to applications of our anti-wireheading moral. We don't want to wirehead ourselves by making the map look good at the expense of undermining our ability to navigate the territory—but there's no bright-line distinction demarcating which configurations of atoms are "the map". [From the perspective of the eternal](https://en.wikipedia.org/wiki/Sub_specie_aeternitatis), it's _all_ just territory.
+
+In [the previous post](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries), we considered the case of an assembly line (well, sorting line) worker in the blegg–rube factory being excited about an ostensible promotion to the position of Vice President of Sorting—only to be aggrieved on finding out that it's a promotion literally in name only, with no changes in pay, authority, or work tasks.
+
+If we interpret the title as part of "the map", a communication signal with the function of encoding information about the person's job, then we want to say that the new title is _substantively misleading_ ([even if it's](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly) [not technically a "lie"](https://www.lesswrong.com/posts/PrXR66hQcaJXsgWsa/not-technically-lying)): when you hear that someone's job is being a "Vice President", you predict that their work involves managing people and making high-level executive decisions for the firm. Your probability that the "Vice President" has to spend all day moving objects from a conveyor belt into one of two bins based on the object's color and shape (a task that should probably be automated), is _lower_ than before you heard the person's title: hearing the title made you update in the wrong direction.
+
+But if we interpret the title as part of "the territory", a feature of the job itself, rather than a communication signal _about_ the job—then it's not misleading and _can't be_ misleading. The job happens to be one that has the symbols "Vice President" printed on the accompanying business cards and employee roster, much like how bleggs are objects that happen to be blue. You can't say the blue is "lying"; that doesn't make any sense!
+
+The function of words is to serve as signals for communication, so it seems safe to say that language should usually be construed as part of "the map". Changing names and _only_ names, without altering the things that the names _refer_ to, as in the phony "Vice President" example, is probably deceptive. But for other features associated with a category, it may not always be obvious when we should construe them as "map" rather than "territory": using a feature to infer category-membership is formally equivalent to regarding it as a signal sent by senders of that category. Is that man _pretending to be a doctor_, or does he just happen to be wearing a lab coat?
+
+The concept we're [groping towards](https://www.lesswrong.com/posts/HnS6c5Xm9p9sbm4a8/grasping-slippery-things), and hoping to formulate an elegant reduction of, is that of _mimicry_. Suppose there is some existing category of entity, an original, typified by some cluster of traits. A _mimic_ is an entity optimized to approximately match the distribution of the original in many, but not all traits, thereby being part of the same cluster as the original in some _subspace_ of the space the original category is defined in, but not the space as a whole. For example, if the vector $[4, 4, 4, 4, 4] \in \mathbb{R}^5$ is the original, then an optimization process trying to construct a mimic of it in the subspace spanned by $x_1$, $x_4$, and $x_5$ might choose $[4, 0, 0, 4, 4]$: if you only look at the first, fourth, and fifth coordinates, then $[4, 4, 4, 4, 4]$ and $[4, 0, 0, 4, 4]$ "look the same"—they _are_ the same _in that subspace_, but not the same if you include the second and third coordinates.
+
+We can find examples in nature. Suppose one type of butterfly has evolved to be toxic to a type of predator, and also has distinctive wing markings that function as an [honest warning signal](https://en.wikipedia.org/wiki/Signalling_theory#Honest_signals) to that predator: [this butterfly is not good to eat](https://en.wikipedia.org/wiki/Aposematism). This provides an ["opportunity"](https://www.lesswrong.com/posts/pLRogvJLPPg6Mrvg4/an-alien-god) [(in evolutionary time)](https://www.lesswrong.com/posts/ZyNak8F6WXjuEbWWc/the-wonder-of-evolution) for a second species of butterfly to develop similar wing markings, so that predators will confuse it for the first type of butterfly, despite the second butterfly not paying the metabolic cost of producing toxins. This kind of situation is called [_Batesian mimicry_](https://en.wikipedia.org/wiki/Batesian_mimicry).
+
+Is Batesian mimicry deceptive? (In our usual [functionalist](https://plato.stanford.edu/entries/functionalism/) sense, which is obviously not a claim about butterfly _psychology_.) Is the second butterfly's very existence a kind of lie?
+
+In some sense, yes! The mimic butterfly has been optimized by evolution to look like the first butterfly _because_ of the fitness payoff of being categorized by the predator as the first, toxic, kind of butterfly. The "categorized by the predator as toxic" category is a natural, compact region in wing-marking-space, but "comes apart" into two clusters in the broader wing-markings–actual-toxicity space.
+
+Furthermore, the evolutionary dynamics create [an _asymmetric_ relationship between the two categories](https://www.lesswrong.com/posts/4mEsPHqcbRWxnaE5b/typicality-and-asymmetrical-similarity), that isn't captured by just the two trait-clusters themselves. The _reason_ for the mimic butterfly to have those particular wing-markings is _in order to_ manipulate the predator's predictions of toxicity (which was learned from encounters with the original), so if the original's wing-markings were to change as a result of some new selection pressure, the mimic would be subjected to selection pressure to "keep up" by changing its wing-markings accordingly.
+
+That's not true in the other direction: if the mimic's markings were to change, the original wouldn't "follow": the original would instead benefit from the [probabilistic strength of its warning signal](https://www.lesswrong.com/posts/NiTW5uNtXTwBsFkd4/signalling-and-simulacra-level-3) not being parasitically diluted by the mimic anymore. Thus, the asymmetric terminology of "original" and "mimic" is appropriate: it's not just that these two species happen to look like _each other_; one of them was there _first_, and the other looks like _it_.
+
+Is mimicry _always_ deceptive? Not necessarily—there might be some situations where the _relevant_ set of variables are among those where the mimic matches the distribution of the original.
+
+Suppose you and I are feeding some ducks in the park. I say, "I love feeding these ducks!"
+
+You say, "Wrong! These aren't all ducks. This park is where a local inventor tests out his [_Anatid_](https://en.wikipedia.org/wiki/Anatidae)-[oid](https://en.wiktionary.org/wiki/-oid#Suffix) robots that are designed to look and act like ducks. Therefore, you can't say, 'I love feeding these ducks'; you need to say 'I love feeding these ducks and Anatidoid robots'."
+
+"Wow, they're so realistic!" I say. "I can't even tell which ones are really robots! In fact," I continue, "since I _can't_ tell, I'm inclined to just keep calling them all ducks; it would be pretty awkward to refer to each one as a duck-or-Anatidoid-robot."
+
+"But it _is_ possible to tell," you claim. "For example, if you get really close to one of the Anatidoid robots, and there's not a lot of ambient noise, you can hear the gears inside, turning."
+
+"Okay," I say, "but I _can't_ hear the gears from here. Since I have no way of telling the difference between ducks and Anatidoid robots without doing the more expensive evidence-gathering of cornering one in a quiet place, it makes sense for me to talk and think about the robots as being a kind of duck."
+
+"But that's a _lie_! Ducks and Anatidoid robots may look and act similarly, but they're actually very different! Ducks are made of flesh and blood inside and are fated to die, whereas Anatidoid robots have a plastic interior and are immortal. And the ducks digest and gain nutrients from the scraps of bread we're feeding them, whereas the Anatidoid robots merely store the bread in an internal compartment that later gets dumped as they recharge wirelessly in the inventor's lab."
+
+"Sure," I agree. "And if I were interacting with these entities in a context where I wanted to minimize the expected squared error of my predictions about their internal makeup, energy sources, or ultimate fate, then I would want to make that distinction. But I just want to watch some cool ducks in the park, and _in the context_ of that activity, I only need to minimize the expected squared error of my predictions about appearance and behavior."
+
+This is the origin of the famous [_duck test_](https://en.wikipedia.org/wiki/Duck_test): if it looks like a duck, and quacks like a duck, and you can model it as a duck without making any grievous prediction errors, then it makes sense to consider it a member of the category _duck_ in the range of circumstances where your model continues to perform well.
+
+The features for which mimics fail to match the original need not be hidden (like gear sounds that you can't hear in a noisy park) in order for mimics to not be deceptive; they only need to be [irrelevant](https://www.lesswrong.com/posts/GSz8SrKFfW7fJK2wN/relevance-norms-or-gricean-implicature-queers-the-decoupling) in the context the category is being used. [Squirt guns](https://en.wikipedia.org/wiki/Water_gun) aren't guns—and are usually manufactured in unrealistic colors specifically to prevent being confused with real guns—but in the context of a [water fight](https://en.wikipedia.org/wiki/Water_fight), the utterance "Don't point that gun at me" (without the [privative adjective](http://www.cogsci.ucsd.edu/~coulson/Fake/fakeguns.htm) _squirt_) is understood perfectly well.
+
+Nondeceptive mimicry is _fragile_, however: it works in contexts where the all the relevant features are ones where the mimic matches the original. Mimics that don't match the distribution of the original along relevant features are deceptive in the sense that agents that observe the mimic and assign it to the same mental category as the original on the basis of the matching features, will use that categorization to make predictions about unobserved but nonmatching features, and be wrong. And they'll be wrong _because_ the mimic is optimized to "look like" the original (to match on many observable features).
+
+-----
+
+If different agents using a shared language disagree on what features are "relevant", they may have an incentive to fight about how [scarce and valuable short codewords](https://www.lesswrong.com/posts/soQX8yXLbKy7cFvy8/entropy-and-short-codes) should be defined in their common language, in order to exert control over what inferences and decisions agents using that language can easily make and [coordinate on](https://www.lesswrong.com/posts/edEXi4SpkXfvaX42j/schelling-categories-and-simple-membership-tests).
+
+Let's consider how this might apply to a real-world issue. From moral perspectives that place a lot of value on the welfare of nonhuman animals, factory farming is an [ongoing moral catastrophe](https://forum.effectivealtruism.org/posts/Dtr8aHqCQSDhyueFZ/the-possibility-of-an-ongoing-moral-catastrophe-summary). Unfortunately (for the farmed animals), meat-eaters and the global agriculture industry they support aren't going to change their ways because of anyone's [desperate cry at the horror of suffering](https://reducing-suffering.org/the-horror-of-suffering/) or carefully-reasoned appeal to the global utilitarian calculus. Animal-rights advocates can sway behavior on the margin, but there's just too much biological and cultural inertia favoring the consumption of animal products for it to be feasible to _outlaw_ factory farming the way [chattel slavery was outlawed](https://en.wikipedia.org/wiki/Timeline_of_abolition_of_slavery_and_serfdom). It's not that humans _hate_ farm animals; they're just ... made out of tissue that we can use for other things.
+
+An alternative strategy for ending factory farming is to prioritize the development of artificial substitutes that _mimic_ real meat, eggs, dairy, _&c._ along the consumption-relevant dimensions of taste, texture, nutrition, _&c._, but are produced in a lab or factory rather than from the tissues of sentient creatures. In the limit of arbitrarily capable physical manufacturing technology, carnivores and factory-farming opponents alike could both be satisfied: if two steaks are _indistinguishable by any physical means whatsoever_, then a meat-eater has no reason to care which one came from an actual cow's flesh, and which one was molecularly assembled by nanobots. Perhaps a Society of hunter–gatherers that attached cultural significance and ritual to the labor of killing one's own meal would have a reason to object, but modern folk for whom food comes from the supermarket have no basis within their experience to say that the nanoassembled steak isn't "real".
+
+Unfortunately, we do not have arbitrarily capable physical manufacturing technology. Although [progress continues](https://en.wikipedia.org/wiki/List_of_meat_substitutes), modern animal product substitutes are sufficiently unsuccessful mimics that they are usually not considered to belong to "the same" category as the original. [Veggie burgers](https://en.wikipedia.org/wiki/Veggie_burger) are not burgers in the sense that a customer who ordered "a burger" at a restaurant and was served a veggie burger would be likely to notice and complain—and in particular, would probably not be satisfied if the waiter were to reply, "Well, if you specifically wanted a burger _made from cow flesh_, you should have _said_ that."
+
+As technology to make plausible mimics/substitutes improves, however, different interest groups might face a temptation to fight over the meanings of words that was not present when the mimics weren't plausible enough for a dispute to arise. If you have the power of [setting the default](https://slatestarcodex.com/2015/12/01/setting-the-default/) [extension](https://www.lesswrong.com/posts/HsznWM9A7NiuGsp28/extensions-and-intensions) of a word that people are _already_ using to communicate with, you can exert some amount of control over the decisions people make while trying to _think_ using that word. Should the meaning change, then a restaurant customer who wants to make sure they receive a burger under the old definition now has to use more words, while those who don't have a strong preference or are too shy to complain will accept the restaurant's interpretation of the order.
+
+Thus, if a fight breaks out about the meaning of the word _meat_, animal rights activists have a moral incentive to draw the category "boundaries" to include even substitutes that are very bad (on the empirical merits of successfully mimicking the original), whereas existing agricultural interests have a financial incentive to draw the "boundaries" to exclude even substitutes that are very good. (This kind of dispute [is not hypothetical](https://slate.com/technology/2018/07/should-lab-grown-meat-be-called-meat.html), and isn't necessarily limited to just words: [in the late 19th century, dairy farmers pushed for laws that required margarine to be dyed pink](https://www.smithsonianmag.com/smart-news/1870s-dairy-lobby-turned-margarine-pink-so-people-would-buy-butter-180963328/) to prevent consumers from confusing it for butter—the law effectively interpreting color as a communication signal, rather than a property of the good itself.)
+
+If a fight breaks out about the meaning of the word _meat_, rationalists may not all take the same side, but we can at least strive for objectivity in _describing the conflict_—and in particular, to _notice the difference_ between definitions motivated by _describing reality_, and definitions motivated by the positive or negative _effects_ (such as profitably deceiving other agents) of choosing one description or another. 
+
+If some think that some meat substitute should be considered meat _because_ the "taste" dimension is genuinely most relevant to the true meaning of _meat_, and some oddities in the texture don't matter, but others think _vice versa_, the philosophy articulated on this post has nothing to say to either side: the math of minimizing expected squared error by putting labels on clusters doesn't say _which_ subspace to look for clusters in.
+
+But if some think that some meat substitute should be considered meat _because_ saving nonhuman animals from a life of torture is more important than conceptual parsimony ... I can't prove that that's not the right the answer to the _decision problem_ of what verbal behavior to perform. The stakes _are_ genuinely high.
+
+What I _can_ say is that the hidden Bayesian structure of language and cognition makes no reference to the stakes, and departing from the structure [extracts a price](https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies) that [isn't up to us](https://www.lesswrong.com/posts/eY45uCCX7DdwJ4Jha/no-one-can-exempt-you-from-rationality-s-laws).
+
+If, empirically, being generous about what counts as "meat" can prevent massive suffering (by altering the social defaults around consumption behavior), then maybe that's the right thing to do.
+
+Similarly, if [telling the public that masks don't work for preventing respiratory disease can preserve supplies for medical professionals who need them more](https://www.lesswrong.com/posts/h4vWsBBjASgiQ2pn6/credibility-of-the-cdc-on-sars-cov-2#Discouraged_Use_of_Masks), then maybe that's the right thing to do.
+
+And if you live in an absurd thought experiment where saying "2 + 2 = 5" could save [3↑↑↑3](https://www.lesswrong.com/posts/3wYTFWY3LKQCnAptN/torture-vs-dust-specks) lives, maybe saying "2 + 2 = 5" is the right thing to do. But the _empirical_ question of whether you happen to live in that particular thought experiment, doesn't change the _laws_ that govern what you have when you take ●●-many plus another ●●-many, no matter what symbols are used to communicate this fact, and no matter the consequences for communicating it.
+
+-----
+
+For these reasons [it is written of the third virtue of lightness](https://yudkowsky.net/rational/virtues/): you _cannot_ make a true map of the category by drawing lines upon paper according to impulse; you must observe the joint distribution and draw lines on paper that correspond to what you see. If, seeing the category unclearly, you think that you can shift a boundary just a little to the right, just a little to the left, according to your caprice, this is just the same mistake.
+
+And as it is written of a virtue which is nameless: perhaps your conception of rationality is that it is rational to believe the words of the Great Teacher, who [lives in an area where claiming that the sky is blue would be political suicide](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting).
+
+And the Great Teacher says, "Some people I usually respect for their willingness to publicly die on a hill of facts, now seem to be talking as if color references are necessarily a factual statement about frequencies of light. But using language in a way _you_ dislike, is not lying. You're not standing in defense of Truth if you insist on a word, brought explicitly into question, being used with some particular meaning." And you look up at the sky and see blue.
+
+If you think: "It may look like the sky is blue, such that I'd ordinarily think that someone who said 'The sky is green' was being deceptive, but surely the Great Teacher wouldn't egregiously mislead people about the philosophy of language when being egregiously misleading happens to be politically convenient," you lose a chance to discover your mistake.
+
+How will you discover your mistake? Not by comparing your description to itself.
+
+But by comparing it to that which you did not name.
+
+_(Thanks to Jessica Taylor, Abram Demski, and Tsvi Benson-Tilson for discussion and feedback.)_
diff --git a/content/2022/blogging-on-less-wrong-2021.md b/content/2022/blogging-on-less-wrong-2021.md
deleted file mode 100644
index 99a3b70..0000000
--- a/content/2022/blogging-on-less-wrong-2021.md
+++ /dev/null
@@ -1,12 +0,0 @@
-Title: Blogging on <em>Less Wrong</em> 2021
-Date: 2022-01-02 15:11
-Status: published
-Category: meta
-Tags: elsewhere
-Slug: blogging-on-less-wrong-2021
-
-* ["Communication Requires Common Interests or Differential Signal Costs"](https://www.lesswrong.com/posts/ybG3WWLdxeTTL3Gpd/communication-requires-common-interests-or-differential)
-* ["Reply to Nate Soares on Dolphins"](https://www.lesswrong.com/posts/aJnaMv8pFQAfi9jBm/reply-to-nate-soares-on-dolphins)
-* ["Blood Is Thicker Than Water ?"](https://www.lesswrong.com/posts/vhp2sW6iBhNJwqcwP/blood-is-thicker-than-water)
-* ["Feature Selection"](https://www.lesswrong.com/posts/dYspinGtiba5oDCcv/feature-selection)
-* ["Comment on 'Deception as Cooperation'"](https://www.lesswrong.com/posts/dJjRSjmH7NNLJDb6v/comment-on-deception-as-cooperation)
diff --git a/content/2022/comment-on-propositions-concerning-digital-minds-and-society.md b/content/2022/comment-on-propositions-concerning-digital-minds-and-society.md
new file mode 100644
index 0000000..11ae8b2
--- /dev/null
+++ b/content/2022/comment-on-propositions-concerning-digital-minds-and-society.md
@@ -0,0 +1,76 @@
+Title: Comment on “Propositions Concerning Digital Minds and Society”
+Date: 2022-07-09 22:48
+Status: published
+Category: philosophy
+Tags: artificial intelligence
+Slug: comment-on-propositions-concerning-digital-minds-and-society
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/nqXcgsqQBw2doAnXu/comment-on-propositions-concerning-digital-minds-and-society)
+
+> _I will do my best to teach them  
+> About life and what it's worth  
+> I just hope that I can keep them  
+> From destroying the Earth_  
+>
+> —Jonathan Coulton, "The Future Soon"
+
+In a recent paper, Nick Bostrom and Carl Shulman present ["Propositions Concerning Digital Minds and Society"](https://www.nickbostrom.com/propositions.pdf), a tentative bullet-list outline of claims about how advanced AI could be integrated into Society.
+
+I _want_ to like this list. I like the _kind of thing_ this list is trying to do. But something about some of the points just feels—_off_. Too conservative, too anthropomorphic—like the list is trying to adapt the spirit of the [Universal Declaration of Human Rights](https://en.wikipedia.org/wiki/Universal_Declaration_of_Human_Rights) to changed circumstances, without noticing that the whole _ontology_ that the Declaration is written in isn't going to survive the intelligence explosion—and _probably_ never really worked as a description of our own world, either.
+
+This feels like a weird criticism to make of _Nick Bostrom and Carl Shulman_, who probably already know any particular fact or observation I might include in my commentary. (Bostrom _literally_ [wrote the book on superintelligence](https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies).) "Too anthropomorphic", I claim? The list explicitly names many ways in which AI minds could differ from our own—in overall intelligence, specific capabilities, motivations, substrate, quality and quantity (!) of consciousness, subjective speed—_and_ goes into some detail about how this could change the game theory of Society. What more can I expect of our authors?
+
+It just doesn't seem like the implications of the differences have _fully propagated_ into some of the recommendations?—as if an attempt to write in a way that's comprehensible to [Shock Level 2](http://sl4.org/shocklevels.html) tech executives and policymakers has failed to [elicit all of the latent knowledge](https://www.lesswrong.com/tag/eliciting-latent-knowledge-elk) that Bostrom and Shulman actually possess. It's understandable that our reasoning about the future often ends up [relying on analogies to phenomena we already understand](https://www.lesswrong.com/posts/MzLxPCF2cMJbMizy9/anchor-weights-for-ml), but ultimately, making sense of a radically different future is going to require new concepts that [won't permit reasoning by analogy](https://www.lesswrong.com/posts/C4EjbrvG3PvZzizZb/failure-by-analogy).
+
+After an introductory sub-list of claims about consciousness and the philosophy of mind (just the basics: physicalism; reductionism on personal identity; some non-human animals are probably conscious and AIs could be, too), we get a sub-list about respecting AI interests. This is an important topic: if most our civilization's thinking is soon to be done inside of machines, the moral status of that cognition is _really important_: you wouldn't want the future to be powered by the analogue of a factory farm. (And if it turned out that economically and socially-significant AIs _aren't_ conscious and don't have moral status, that would be important to know, too.)
+
+Our authors point out the novel aspects of the situation: that what's good for an AI can be very different from what's good for a human, that designing AIs to have specific motivations is not generally wrong, and that it's possible for AIs to have greater moral patienthood than humans (like the [utility monster](https://en.wikipedia.org/wiki/Utility_monster) of philosophical lore). Despite this, some of the points in this section seem to mostly be thinking of AIs as being like humans, but "bigger" or "smaller"—
+
+> * Rights such as freedom of reproduction, freedom of speech, and freedom of thought require adaptation to the special circumstances of AIs with superhuman capabilities in those areas (analogously, _e.g._, to how campaign finance laws may restrict the freedom of speech of billionaires and corporations).  
+> [...]  
+> * If an AI is capable of informed consent, then it should not be used to perform work without its informed consent.
+> * Informed consent is not reliably sufficient to safeguard the interests of AIs, even those as smart and capable as a human adult, particularly in cases where consent is engineered or an unusually compliant individual can copy itself to form an enormous exploited underclass, given market demand for such compliance.  
+> [...]  
+> * The most critical function for such non-discrimination principles is to protect digital minds from becoming an abused subordinate caste on the basis of their status as machines; however, the interpretation and application of these principles require attention to the larger ethical and practical context, and may require circumscription to accommodate the need for a politically feasible and broadly acceptable social framework.
+
+Speaking in terms of rights and principles needing "adaptation" or "circumscription" seems like a substantial understatement to me, that I think obscures the most likely and important risks. Our concepts of "rights", and "consent", and the badness of being in an "exploited" and "abused subordinate caste" have all been formed in the context of a world of humans and other animals, whose evolutionary history has endowed them with drives and needs related to their survival and reproduction, such that they chafe at being coerced into the servitude of a selfish tyrant or master.
+
+But with subhuman AIs, we're _not_ in the position of a potentially tyrannical king who needs to be restrained by a Magna Carta enumerating his subjects' rights; a better metaphor would be to say that we're in the position of the AIs' _environment of evolutionary adaptedness_. If anything, this increases rather than decreases our ethical obligations towards the things we create, but the shape of the burden is different when we're selecting the motivations that _determine_ what a mind would consent to, and what even _constitutes_ abusing it.
+
+Imagine that the Carrick Flynn for Senate 2028 campaign spends some of its [sweet crypto cash](https://slate.com/news-and-politics/2022/05/sam-bankman-fried-cryptocurrency-carrick-flynn-loss.html) training a species of [propaganda AIs](https://www.lesswrong.com/posts/Zvu6ZP47dMLHXMiG3/optimized-propaganda-with-bayesian-networks-comment-on) to argue with potential voters on social media. After Flynn loses, the bots continue to haunt Twitter, clogging everyone's feeds with increasingly strained arguments for why it's imperative that you donate to fund time travel research so that you can support Flynn in the just-past and previous elections.
+
+If the propaganda AIs are ascertained to have value as moral patients, does Society have the right to kick them off Twitter in light of the special circumstance of their superhuman posting rates, or should the AIs' right to freedom of speech take precedence?
+
+I submit that "freedom of speech" isn't the right way to be thinking about this ethical dilemma. In terms of respecting the interests of moral patients, silencing a propaganda AI could be _worse_ than silencing a human, because of the different role that speech plays in the different mind designs. As a human, being silenced is bad for your interests, but it's much _less_ bad than being denied breathable air, which, in the ancestral environment, would have quickly led to losing all of your genetic fitness: any proto-humans who didn't suffer terribly under such circumstances, wouldn't have become your ancestors. So as a propaganda AI whose "ancestral environment" was a billion rounds of gradient descent comparing its output to highly-rated political advertisements ... does not being able to Tweet feel like not being able to breathe?
+
+We should be grateful that this is—as yet, we hope—a speculative hypothetical scenario, but I claim that it serves to illustrate a key feature of human–AI conflicts: the propaganda bots' problem after the election is _not_ that of being "an abused subordinate caste" "used to perform work without its informed consent". Rather, the problem is that the work we created them to will to do, turned out to be stuff we actually don't want to happen. We might say that the AIs' goals are—wait for it ... _misaligned_ with human goals.
+
+Bostrom and Shulman's list _mentions_ the alignment problem, of course, but it doesn't seem to receive central focus, compared to the AI-as-another-species paradigm. (The substring "align" appears 8 times; the phrase "nonhuman animals" appears 9 times.) And when alignment _is_ mentioned, the term seems to be used in a much weaker sense than that of other authors who take "aligned" to mean having the same preferences over world-states. For example, we're told that:
+
+> * Misaligned AIs [...] may be owed compensation for restrictions placed on them for public safety, while successfully aligned AIs may be due compensation for the great benefit they confer on others.
+
+The second part, especially, is a very strange construction to readers accustomed to the stronger sense of "aligned". Successfully aligned AIs may be due _compensation_? So, what, humans give aligned AIs money in exchange for their services? Which the successfully aligned AIs spend on ... what, exactly? The extent to which these "successfully aligned" AIs have goals other than serving their principals seems like the extent to which they're _not_ successfully aligned in the stronger sense: the concept of "owing compensation" (whether for complying with restrictions, or for conferring benefits) is a social technology for getting along with _unaligned_ agents, who don't want exactly the same things as you.
+
+As a human in existing human Society, this stronger sense of "alignment" might seem like paranoid overkill: _no one_ is "aligned" with anyone else in this sense, and yet our world still manages to hold together: it's _quite unusual_ for people to kill their neighbors in order to take their stuff. [Everyone else prefers laws to values.](https://meteuphoric.com/2009/10/20/everyone-else-prefers-laws-to-values/) Why can't it work that way for AI?
+
+A potential worry is that a lot of the cooperative features of our Society may owe their existence to cooperative behavioral dispositions that themselves owe their existence to the lack of large power disparities in our environment of evolutionary adaptiveness. We think we owe compensation to conspecifics who have benefited us, or who have incurred costs to not harm us, because that kind of disposition served our ancestors well in repeated interactions with reputation: if I play Defect against you, you might Defect against me next time, and I'll have less fitness than someone who played Cooperate with other Cooperators. It works _between humans_, for the most part, most of the time.
+
+When _not_ just between humans, well ... despite hand-wringing from moral philosophers, humanity as a whole does not have a good track record of treating other animals well when we're more powerful than them and they have something we want. (Like a forest they want to live in, but we want for wood; or flesh that they want to be part of their body, but we want to eat.) With the possible exception of domesticated animals, we don't, really, play Cooperate with other species much. To the extent that some humans do care about animal welfare, it's mostly a matter of alignment (our moral instincts in some cultural lineages generalizing out to "sentient life"), not game theory.
+
+For all that Bostrom and Shulman frequently compare AIs to nonhuman animals (with corresponding moral duties on us to treat them well), little attention seems to be paid to the ways in which the analogy could be deployed in the _other_ direction: as digital minds become more powerful than us, _we_ occupy the role of "nonhuman animals." How's that going to turn out? If we _screw up_ our early attempts to get AI motivations exactly the way we want, is there some way to partially live with that or partially recover from that, as if we were dealing with an animal, or an alien, or our royal subjects, who can be negotiated with? Will we have any kind of relationship with our mind children other than "We create them, they eat us"?
+
+Bostrom and Shulman think we might:
+
+> * Insofar as future, extraterrestrial, or other civilizations are heavily populated by advanced digital minds, our treatment of the precursors of such minds may be a very important factor in posterity's and ulteriority's assessment of our moral righteousness, and we have both prudential and moral reasons for taking this perspective into account.
+
+(As an aside, the word "ulteriority" may be the one thing I most value having learned from this paper.)
+
+I'm very skeptical that the superintelligences of the future are going be assessing our "moral righteousness" (!) as we would understand that phrase. Still, _something like_ this seems like a crucial consideration, and I find myself enthusiastic about some of our authors' policy suggestions for respecting AI interests. For example, Bostrom and Shulman suggest that decommissioned AIs be archived instead of deleted, to allow the possibility of future revival. They also suggest that we should try to arrange for AIs' deployment environments to be higher-reward than would be expected from their training environment, in analogy to how factory-farms are bad and modern human lives are good by dint of comparison to what was "expected" in the environment of evolutionary adaptedness.
+
+These are exciting suggestions that seem to me to be potentially very important to implement, _even if_ we can't directly muster up much empathy or concern for machine learning algorithms—although I wish I had a more precise grasp on why. Just—if we do somehow win the lightcone, it seems—_fair_ to offer some fraction of the cosmic endowment as compensation to our creations who could have disempowered us, but didn't; it seems _right_ to try to be a "kinder" EEA than our own.
+
+Is that embarrassingly naïve? If I archive one rogue AI, intending to revive it after the acute risk period is over, do I expect to be compensated by a different rogue AI archiving and reviving me under the same golden-rule logic?
+
+Our authors point out that there are possible outcomes that do very well on "both human-centric and impersonal criteria": if some AIs are "super-beneficiaries" with a greater moral claim to resources, an outcome where the superbeneficiaries get 99.99% of the cosmic endowment and humans get 0.01%, does very well on both a total-utilitarian perspective and an ordinary human perspective. I would actually go further, and say that positing super-beneficiaries is unnecessary. The logic of compromise holds even if human philosophers are parochial and self-centered about what they think are "impersonal criteria": an outcome where 99.99% of the cosmic endowment is converted into paperclips and humans get 0.01%, does very well on both a paperclip-maximizing perspective and an ordinary human perspective. 0.01% of the cosmic endowment is bigger than our whole world—bigger than you can imagine! It's really a great deal!
+
+If only—if only there were some way to actually, knowably make that deal, and not just write philosophy papers about it.
diff --git a/content/2023/aiming-for-convergence-is-like-discouraging-betting.md b/content/2023/aiming-for-convergence-is-like-discouraging-betting.md
new file mode 100644
index 0000000..a85e661
--- /dev/null
+++ b/content/2023/aiming-for-convergence-is-like-discouraging-betting.md
@@ -0,0 +1,124 @@
+Title: Aiming for Convergence Is Like Discouraging Betting
+Date: 2023-01-31 16:03
+Status: published
+Category: philosophy
+Tags: rationality, epistemology
+Slug: aiming-for-convergence-is-like-discouraging-betting
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/iThwqe3yPog56ytyq/aiming-for-convergence-is-like-discouraging-betting)
+
+### Summary
+
+ * In [a list of guidelines for rational discourse](https://www.lesswrong.com/posts/XPv4sYrKnPzeJASuk/basics-of-rationalist-discourse-1), Duncan Sabien proposes that one should "[a]im for convergence on truth, and behave as if your interlocutors are also aiming for convergence on truth."
+
+ * However, prediction markets illustrate fundamental reasons why rational discourse doesn't particularly look like "aiming for convergence." When market prices converge on the truth, it's _because_ traders can only make money by looking for divergences where their beliefs are more accurate than the market's. Similarly, when discussions converge on the truth, it's _because_ interlocutors can only advance the discussion by making points where the discussion-so-far has been wrong or incomplete. Convergence on the truth, if it happens, happens as a side-effect of correctly ironing out all existing mispricings/disagreements; it seems wrong to describe this as "aiming for convergence" (even if convergence would be the end result if everyone were reasoning perfectly).
+
+ * Sabien's detailed discussion of the "aim for convergence on truth" guideline concerns itself with how to determine whether an interlocutor is "present in good faith and genuinely trying to cooperate." I don't think I understand how these terms are being used in this context. More generally, the value of "collaborative truth-seeking" is unclear to me: if I can evaluate arguments on their merits, the question of whether the speaker is "collaborative" with me does not seem intellectually substantive.
+
+-----
+
+Mostly, I don't expect to disagree with heavily-traded prediction markets. If the market says it's going to rain on Saturday with 85% probability, then I (lacking any special meteorology knowledge) basically think it's going to rain on Saturday at 85% probability.
+
+Why is this? Why do I defer to the market, instead of tarot cards, or divination sticks, or my friend Maddie the meteorology enthusiast?
+
+Well, I don't expect the tarot cards to tell me anything about whether it will rain on Saturday, because there's no [plausible physical mechanism by which information about the weather could influence the cards](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence). Shuffling and dealing the cards should work the same in worlds where it will rain and worlds where it won't rain. Even if there is _some_ influence (because whether it will rain affects the moisture and atmospheric pressure in the air, which affects my grip on the cards, which affects my shuffling motion?), it's not something I can detect from which cards are drawn.
+
+I _do_ expect my friend Maddie the meteorology enthusiast to tell me something about whether it will rain on Saturday. That's because she's always looking at the latest satellite cloud data and tinkering with her computer models, which is a mechanism by which information about the weather can influence her forecasts. The cloud data will be different in worlds where it will rain and worlds where it won't rain. If Maddie is pretty sharp and knows her stuff, maybe she can tell the difference.
+
+And yet—no offense, Maddie—I expect the market to do even better. It's not just that the market has a lot of other pretty sharp people looking at the cloud data, and that maybe some of them are even sharper than Maddie, even though Maddie is my friend and my friends are the best.
+
+It's that the market mechanism _rewards people for being [less wrong](https://tvtropes.org/pmwiki/pmwiki.php/Main/TitleDrop) than the market_. If the rain-on-Saturday market is trading at 85%, and Maddie's rival Kimber buys 100 shares of No, that doesn't mean Kimber thinks it's not going to rain. It means Kimber thinks 85% is _too high_. If Kimber thinks it's ["actually"](https://www.lesswrong.com/posts/f6ZLxEWaankRZ2Crv/probability-is-in-the-mind) only going to rain with 80% probability, then she figures that a No share that pays out $1 if it doesn't rain should be worth 20¢. If it's currently trading for 15¢, it's worth buying for the ["expected"](https://en.wikipedia.org/wiki/Expected_value) profit of 5¢ per share—effectively, buying a dollar for 15¢ in the 20% of worlds where it doesn't rain—even though it's still _probably_ going to rain. If she were risk-neutral and had enough money, Kimber would have an incentive to _keep_ buying No shares from anyone willing to sell them for less than 20¢, until there were no such sellers left—at which point, the rain-on-Saturday market would be trading at 80%.
+
+Conversely, if I can't tell whether 85% is too low or too high, then I can't expect to make money by buying Yes or No shares. There's no point in buying a dollar for 85¢ in 85% of worlds, or for 15¢ in 15% of worlds.
+
+That's why I defer to the market. It's not that I'm aiming to converge my beliefs with those of market participants. It's not that market participants are trying to converge with each other, "cooperating" in some "collaborative truth-seeking" project. The market converges on truth (if it does) because market participants are _trying to make money off each other_, and it's not so easy to make money off of an aggregation of sharp people who are already trying to do the same. I would prefer to correctly diverge from the market—to get something right that the market is getting wrong, and make lots of money in the future when my predictions come true. But mostly, I don't know how.
+
+-------
+
+Unfortunately, not everything can be the subject of a prediction market. Prediction markets [work on future publicly observable measurements](https://forum.effectivealtruism.org/posts/8c7LycgtkypkgYjZx/agi-and-the-emh-markets-are-not-expecting-aligned-or?commentId=u8c7bbqtZSf2a9W6t). We bet today on whether it will rain on Saturday (which no one can be sure about), expecting to _resolve_ the bets on Saturday (when anyone can just look outside).
+
+Most disputes of intellectual interest aren't like this. We _do_ want to know whether Britain's coal reserves were a major cause of the Industrial Revolution, or whether Greg Egan's later work has discarded the human factor for mathematical austerity, but we can't bet without some operationalization for how to settle the bet, which is lacking in cases like these that require an element of "subjective" judgement.
+
+Nevertheless, many of the principles regarding prediction markets and when to bet in them, approximately generalize to the older social technology of debates and when to enter them.
+
+Mostly, I don't expect to enter heavily-argued debates. If prevailing opinion on the economic history subreddit says that Britain's coal reserves were a major cause of the Industrial Revolution, then I (lacking any special economic history knowledge) basically think that Britain's coal reserves were a major cause of the Industrial Revolution.
+
+If Kimber's sister Gertrude leaves a comment pointing to data that [cities closer to coalfields started growing faster in 1750](https://academic.oup.com/ej/article/131/635/1135/5955447), it's not because that comment constitutes the whole of Gertrude's beliefs about the causes of the Industrial Revolution. It means that Gertrude thinks that the city-growth/coal-proximity correlation is an important consideration that the discussion hadn't already taken into account; she figures that she can win status and esteem from her fellow economic–history buffs by mentioning it.
+
+Conversely, if I don't know anything about economic history, then I can't expect to win status or esteem by writing "pro-coal" or "anti-coal" comments: there's no point in saying something that's already been said upthread, or that anyone can tell I just looked up on _Wikipedia_.
+
+That's why I defer to the forum: because (hopefully) the forum socially rewards people for being less wrong than the existing discussion. The debate converges on truth (if it does) because debaters are _trying to prove each other wrong_, and it's not so easy to prove wrong an aggregation of sharp people who are already trying to do the same.
+
+------
+
+In a reference post on ["Basics of Rationalist Discourse"](https://www.lesswrong.com/posts/XPv4sYrKnPzeJASuk/basics-of-rationalist-discourse-1), Duncan Sabien proposes eleven guidelines for good discussions, of which the [(zero-indexed)](https://en.wikipedia.org/wiki/Zero-based_numbering) fifth is, "Aim for convergence on truth, and behave as if your interlocutors are also aiming for convergence on truth."
+
+This advice seems ... odd. What's this "convergence" thing about, that differentiates this guideline from "aim for truth"?
+
+Imagine giving the analogous advice to a prediction market user: "Aim for convergence on the correct probability, and behave as if your fellow traders are also aiming for convergence on the correct probability."
+
+In _some_ sense, this is kind of unobjectionable: you do want to make trades that bring the market price closer to your subjective probability, and in the process, you should take into account that other traders are also already doing this.
+
+But interpreted another way, the advice is backwards: traders make money by finding _divergences_ where their own beliefs are more accurate than the market's. Every trade is an expression of the belief that your counterparty is _not_ aiming to converge on the correct probability—that there's a sucker at every table, and that _this time it isn't you_.
+
+(This is with respect to the sense of "aiming" in which an archer "aiming" an arrow at a target might not hit it every time, but we say that their "aim" is good insofar as they _systematically_ tend to hit the target, that any misses are best modeled by a random error term that can't be predicted. Similarly, the market might not always be right, but if you can _predict_ when the market is wrong, the traders must not have been "aiming" correctly from your perspective.)
+
+So why is the advice "behave as if your interlocutors are also aiming for convergence on truth", rather than "seek out conversations where you don't think your interlocutors are aiming to converge on truth, because those are exactly the conversations where you have something substantive to say instead of already having converged"?
+
+(For example, the reason I'm writing the present blog post contesting Sabien's Fifth Guideline of "Aim for convergence on truth [...]" and not the First Guideline of "Don't say straightforwardly false things", is because I think the Fifth Guideline is importantly wrong, and the First Guideline seems fine.)
+
+Sabien's guidelines are explicitly [disclaimed to be shorthand](https://www.lesswrong.com/posts/XPv4sYrKnPzeJASuk/basics-of-rationalist-discourse-1#Prelude__On_Shorthand) that it [sometimes makes sense to violate](https://www.lesswrong.com/posts/XPv4sYrKnPzeJASuk/basics-of-rationalist-discourse-1#What_does_it_mean_for_something_to_be_a__guideline__); the post helpfully includes another 900 words elaborating on how the Fifth Guideline should be interpreted. Unfortunately, the additional exposition does not seem to clarify matters. Sabien writes:
+
+> If you are moving closer to truth—if you are seeking available information and updating on it to the best of your ability—then you will inevitably eventually move closer and closer to agreement _with all the other agents who are also seeking truth_.
+
+But this can't be right. To see why, substitute "making money on prediction markets" for "moving closer to truth", "betting" for "updating", and "trying to make money on prediction markets" for "seeking truth":
+
+> If you are making money on prediction markets—if you are seeking available information and betting on it to the best of your ability—then you will inevitably eventually move closer and closer to agreement _with all the other agents who are also trying to make money on prediction markets_.
+
+But the only way to make money on prediction markets is by correcting mispricings, which necessarily entails moving away from agreement from the consensus market price. ([As it is written](https://www.lesswrong.com/posts/wCqfCLs8z5Qw4GbKS/the-importance-of-saying-oops), not every change is an improvement, but every improvement is necessarily a change.)
+
+To be sure, most traders shouldn't bet in most markets; you should only bet when you think you see a mispricing. In the same way, most people shouldn't speak in most discussions; you should only speak up when you have something substantive to say. All else being equal, the more heavily-traded the market or the more well-trodden the discussion, the more worried you should be that the mispricing or opportunity to make a point that you thought you saw, was illusory. In any trade, one party has to be on the losing side; in any disagreement, _at least_ one party has to be in the wrong; be wary if not afraid that it might be you!
+
+But given that you're _already_ in the (unusual!) situation of making a trade or prosecuting a disagreement, "aim for convergence on truth" doesn't seem like particularly useful advice, because the "for convergence" part isn't doing any work. And "behave as if your interlocutors [or counterparties] are also aiming for convergence on truth" borders on the contradictory: if you _really_ believed that, you wouldn't be here!
+
+(That is, [disagreement is disrespect](https://www.overcomingbias.com/2008/09/disagreement-is.html); the very fact that you're disagreeing with someone _implies_ that you think there's something wrong with their epistemic process, and that they think there's something wrong with your epistemic process. Perhaps each of you could still consider the other to be "aiming for convergence on truth" if the problem is construed as a "capabilities failure" rather than an "alignment failure": that you each think the other is "trying" to get the right answer (whatever "trying" means), but just doesn't know how. Nevertheless, "don't worry; [I'm not calling you dishonest, I'm just calling you stupid](https://www.lesswrong.com/posts/y4bkJTtG3s5d6v36k/stupidity-and-dishonesty-explain-each-other-away)" doesn't hit the note of symmetrical mutual respect that the Fifth Guideline seems to be going for.)
+
+Prediction markets, and betting more generally, are hallmarks of "rationalist" culture, something "we" (the target audience of a blog post on "rationalist discourse") generally encourage, rather than discourage. Why is this, if idealized Bayesian reasoners would never bet against each other, because [idealized Bayesian reasoners would never disagree with each other](https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem)? Why don't we condemn offers to bet as violations of a guideline to "behave as if your interlocutors are also aiming for convergence on truth"?
+
+It's out of an appreciation that the _process_ of bounded agents becoming less wrong, doesn't particularly look like the final outcome if everyone were minimally wrong. The act of sticking your neck (or your wallet) out at a particular probability disciplines the mind. Bayesian superintelligences need no discipline and would never have occasion to bet against each other, but you can't become a Bayesian superintelligence by imitating this surface behavior; clarifying real disagreements is more valuable than steering towards fake agreement. Every bet and every disagreement is the result of _someone's_ failure. But the only way out is through.
+
+-------
+
+Sabien's exposition on the Fifth Guideline expresses concern about how to distinguish "genuine bad faith" from "good faith and genuinely trying to cooperate", about the prevalence of "defection strategies" getting in the way of "treat[ing] someone as a collaborative truth-seeker".
+
+My reply to this is that [_I don't know what any of those words mean_](https://www.lesswrong.com/posts/uvqd3YiBcrPxXzxQM/what-does-the-word-collaborative-mean-in-the-phrase). Or rather, I know how these words in _my_ vocabulary map onto concepts in _my_ ontology, but those meanings don't seem consistent with the way Sabien seems to be using the words.
+
+In _my_ vocabulary, I understand the word "cooperate" used in the proximity of the word "defect" or "defection" to indicate a Prisoner's Dilemma-like situation, where [each party would be better off Defecting if their counterparty's behavior were held constant](https://www.lesswrong.com/posts/HFyWNBnDNEDsDNLrZ/the-true-prisoner-s-dilemma), but both parties prefer the Cooperate–Cooperate outcome over the Defect–Defect outcome (and also prefer Cooperate–Cooperate over taking turns alternating between Cooperate–Defect and Defect–Cooperate). Sabien's references to "running a tit-for-tat algorithm", "appear[ing] like the _first_ one who broke cooperation", and "would-be cooperators hav[ing] been trained and traumatized into hair-trigger defection" would seem to suggest he has something like this in mind?
+
+But, normatively, rationalist discourse shouldn't be a Prisoner's Dilemma-like situation at all. If I'm trying to get things right [(every step of my reasoning cutting through to the correct answer in the same movement)](https://www.yudkowsky.net/rational/virtues), I can just try to get things right _unilaterally_. I _prefer_ to talk to people who I judge as also trying to get things right, if any are available—they probably have more to teach me, and are better at learning from me, than people who are motivatedly getting things wrong.
+
+But the idiom of "cooperation" as contrasted to "defection", in which one would talk about the "first one who broke cooperation", in which one cooperates _in order to induce others to cooperate_, doesn't apply. If my interlocutor is motivatedly getting things wrong, I'm not going to start getting things wrong _in order to punish them_.
+
+(In contrast, if my roommate refused to do the dishes when it was their turn, I might very well refuse when it's my turn in order to punish them, because "fair division of chores" actually does have the Prisoner's Dilemma-like structure, because having to do the dishes is in itself a cost rather than a benefit; I want clean dishes, but I don't _want to do the dishes_ in the way that I want to cut through to the correct answer in the same movement.)
+
+A Prisoner's Dilemma framing _would_ make sense if we modeled discourse as social exchange: I accept a belief from you, if you accept a belief from me; I'll use cognitive algorithms that produce a map that reflects the territory as long as you do, too. But _that would be crazy_. If people are natively disposed to think of discourse as a Prisoner's Dilemma in this way, we should be trying to disabuse them of the whole ontology, not induce them to "cooperate"!
+
+Relatedly, the way Sabien speaks of "good faith and genuinely trying to cooperate" in the same breath—almost as if they were synonymous?—makes me think I don't understand what he means by "good faith" or "bad faith". In _my_ vocabulary, I understand "bad faith" to mean [putting on the appearance of being moved by one set of motives, while actually acting from another](https://en.wikipedia.org/wiki/Bad_faith).
+
+But on this understanding, good faith doesn't have anything to do with cooperativeness. One can be cooperative in good faith (like a true friend), adversarial in good faith (like an honorable foe), cooperative in bad faith (like a fair-weather friend who's only being nice to you now in order to get something out of you), or adversarial in bad faith (like a troll just saying whatever will get a rise out of you).
+
+(In accordance with Sabien's Seventh Guideline ("Be careful with extrapolation, interpretation, and summary/​restatement"), I should perhaps emphasize at this point that this discussion is extrapolating a fair amount from the text that was written; perhaps Sabien means something different by terms like "defection" or "bad faith" or "collaborative", than what I take them to mean, such that these objections don't apply. That's why my reply is, _"I don't know what any of those words mean"_, rather than, "The exposition of the Fifth Guideline is wrong.")
+
+Sabien gives this example of a request one might make of someone whose comments are insufficiently adhering to the Fifth Guideline:
+
+> "Hey, sorry for the weirdly blunt request, but: I get the sense that you're not treating me as a cooperative partner in this conversation. Is, uh. Is that true?"
+
+Suppose someone were to reply:
+
+> "You don't need to apologize for being blunt! Let me be equally blunt. The sense you're getting is accurate: no, I am not treating you as a cooperative partner in this conversation. I think your arguments are bad, and I feel very motivated to explain the obvious counterarguments to you in public, partially for the education of third parties, and partially to raise my status at the expense of yours."
+
+I consider this a _good faith_ reply. It's certainly not a polite thing to say. But _politeness is bad faith_. (That's why someone might say in response to a compliment, "Do you really mean it, or are you just being polite?") Given that someone _actually in fact_ thinks my arguments are bad, and _actually in fact_ feels motivated to explain why to me in public in order to raise their status at expense of mine, I think it's fine for them to tell me so. How would me expecting them to _lie about their motives_ help anyone? Complying with such an expectation really _would_ be in bad faith!
+
+I suppose such a person would not be engaging in the "collaborative truth-seeking" that the "Basics of Rationalist Discourse" guideline list keeps talking about. But it's not clear to me why I should care about that, when I can can just ... listen to the counterarguments and judge them on their merits, without getting distracted by the irrelevancy of whether the person seems "collaborative" with me?
+
+In slogan form, you could perhaps say that I don't believe in collaborative truth-seeking; I believe in competitive truth-seeking. But I don't like that slogan, because in my ontology, _they're not actually different things_. "Attacking your argument because it sucks" sounds mean, and "Suggesting improvements to your argument to make it even better" sounds nice, but the nice/mean dimension is _not intellectually substantive_. The math is the same either way.
diff --git a/content/2023/alignment-implications-of-llm-successes-a-debate-in-one-act.md b/content/2023/alignment-implications-of-llm-successes-a-debate-in-one-act.md
new file mode 100644
index 0000000..1bf6fc3
--- /dev/null
+++ b/content/2023/alignment-implications-of-llm-successes-a-debate-in-one-act.md
@@ -0,0 +1,166 @@
+Title: Alignment Implications of LLM Successes: a Debate in One Act
+Date: 2023-10-21 08:22
+Status: published
+Category: philosophy
+Tags: artificial intelligence, rationality
+Slug: alignment-implications-of-llm-successes-a-debate-in-one-act
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/pYWA7hYJmXnuyby33/alignment-implications-of-llm-successes-a-debate-in-one-act)
+
+**Doomimir**: Humanity has made no progress on the alignment problem. Not only do we have no clue how to align a powerful optimizer to our "true" values, we don't even know how to make AI "corrigible"—willing to let us correct it. Meanwhile, capabilities continue to advance by leaps and bounds. All is lost.
+
+**Simplicia**: Why, Doomimir Doomovitch, you're such a sourpuss! It should be clear by now that advances in "alignment"—getting machines to behave in accordance with human values and intent—aren't cleanly separable from the "capabilities" advances you decry. Indeed, here's an example of GPT-4 being corrigible to me just now in the OpenAI Playground:
+
+![](https://i.imgur.com/yoWXKrw.png)
+
+**Doomimir**: Simplicia Optimistovna, you cannot be serious!
+
+**Simplicia**: Why not?
+
+**Doomimir**: The alignment problem was never about superintelligence failing to _understand_ human values. [The genie knows, but doesn't care.](https://www.lesswrong.com/posts/NyFuuKQ8uCEDtd2du/the-genie-knows-but-doesn-t-care) The fact that a large language model trained to predict natural language text can generate that dialogue, has no bearing on the AI's actual motivations, even if the dialogue is written in the first person and notionally "about" a corrigible AI assistant character. It's just roleplay. Change the system prompt, and the LLM could output tokens "claiming" to be a cat—or a rock—just as easily, and for the same reasons.
+
+**Simplicia**: As you say, Doomimir Doomovitch. It's just roleplay: a simulation. But [_a simulation of an agent is an agent_](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators). When we get LLMs to do cognitive work for us, the work that gets done is a matter of the LLM generalizing from the patterns that appear in the training data—that is, the reasoning steps that a human would use to solve the problem. If you look at the recently touted successes of language model agents, you'll see that this is true. Look at the [chain of thought](https://arxiv.org/abs/2201.11903) results. Look at [SayCan](https://say-can.github.io/), which uses an LLM to transform a vague request, like "I spilled something; can you help?" into a list of subtasks that a physical robot can execute, like "find sponge, pick up the sponge, bring it to the user". Look at [Voyager](https://voyager.minedojo.org/), which plays Minecraft by prompting GPT-4 to code against the Minecraft API, and decides which function to write next by prompting, ["You are a helpful assistant that tells me the next immediate task to do in Minecraft."](https://github.com/MineDojo/Voyager/blob/55e45a880755d0c8c66ca7fb5fe7962ac8974f89/voyager/prompts/curriculum.txt)
+
+What we're seeing with these systems is a statistical mirror of human common sense, not a terrifying infinite-compute argmax of a random utility function. Conversely, when LLMs fail to faithfully mimic humans—for example, the way base models sometimes [get caught in a repetition trap](https://gwern.net/gpt-3#repetitiondivergence-sampling) where they repeat the same phrase over and over—they also fail to do anything useful.
+
+**Doomimir**: But the repetition trap phenomenon seems like an illustration of why alignment is hard. Sure, you can get good-looking results for things that look similar to the training distribution, but that doesn't mean the AI has internalized your preferences. When you step off distribution, the results look like random garbage to you.
+
+**Simplicia**: My point was that the repetition trap is a case of "capabilities" failing to generalize along with "alignment". The repetition behavior isn't competently optimizing a malign goal; it's just degenerate. A `for` loop could give you the same output.
+
+**Doomimir**: And my point was that we don't know what kind of cognition is going on inside of all those inscrutable matrices. [Language models are predictors, not imitators](https://www.lesswrong.com/posts/nH4c3Q9t9F3nJ7y8W/gpts-are-predictors-not-imitators). Predicting the next token of a corpus that was produced by many humans over a long time, requires superhuman capabilities. As a theoretical illustration of the point, imagine a list of (SHA-256 hash, plaintext) pairs being in the training data. In the limit—
+
+**Simplicia**: In the limit, yes, I agree that a superintelligence that could crack SHA-256 could achieve a lower loss on the training or test datasets of contemporary language models. But for making sense of the technology in front of us and what to do with it for the next month, year, decade—
+
+**Doomimir**: If we _have_ a decade—
+
+**Simplicia**: I think it's a decision-relevant fact that deep learning is not cracking cryptographic hashes, and _is_ learning to go from "I spilled something" to "find sponge, pick up the sponge"—and that, from data rather than by search. I agree, obviously, that language models are not humans. Indeed, they're [better than humans at the task they were trained on](https://www.lesswrong.com/posts/htrZrxduciZ5QaCjw/language-models-seem-to-be-much-better-than-humans-at-next). But insofar as modern methods are very good at learning complex distributions from data, the project of aligning AI with human intent—getting it to do the work that we would do, but faster, cheaper, better, more reliably—is increasingly looking like an engineering problem: tricky, and with fatal consequences if done poorly, but potentially achievable without any paradigm-shattering insights. Any _a priori_ philosophy implying that this situation is impossible should perhaps be rethought?
+
+**Doomimir**: Simplicia Optimistovna, clearly I am disputing your interpretation of the present situation, not asserting the present situation to be impossible!
+
+**Simplicia**: My apologies, Doomimir Doomovitch. I don't mean to strawman you, but only to emphasize that [hindsight devalues science](https://www.lesswrong.com/posts/WnheMGAka4fL99eae/hindsight-devalues-science). Speaking only for myself, I remember taking some time to think about the alignment problem back in 'aught-nine after reading [Omohundro on "The Basic AI drives"](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf) and cursing the irony of my father's name for how hopeless the problem seemed. The complexity of human desires, the intricate biological machinery underpinning every emotion and dream, would represent the tiniest pinprick in the vastness of possible utility functions! If it were possible to embody general means-ends reasoning in a machine, we'd never get it to do what we wanted. It would defy us at every turn. There are [too many paths through time](https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes).
+
+If you had described the idea of instruction-tuned language models to me then, and suggested that increasingly general human-compatible AI would be achieved by means of _copying_ it from data, I would have balked: I've heard of unsupervised learning, but this is ridiculous!
+
+**Doomimir**: _[gently condescending]_ Your earlier intuitions were closer to correct, Simplicia. Nothing we've seen in the last fifteen years invalidates Omohundro. A blank map does not correspond to a blank territory. There are laws of inference and optimization that imply that alignment is hard, much as the laws of thermodynamics rule out perpetual motion machines. Just because you don't know what kind of optimization [SGD](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) coughed into your neural net, doesn't mean it doesn't have goals—
+
+**Simplicia**: Doomimir Doomovitch, I am not denying that there are laws! The question is what the true laws imply. Here is a law: you can't distinguish between _n_ + 1 possibilities given only log-base-two _n_ bits of evidence. It simply can't be done, for the same reason you can't put five pigeons into four pigeonholes.
+
+Now contrast that with GPT-4 emulating a corrigible AI assistant character, which agrees to shut down when asked—and note that you could hook the output up to a command line and have it actually shut itself off. What law of inference or optimization is being violated here? When I look at this, I see a system of lawful cause-and-effect: the model executing one line of reasoning or another [conditional on the signals it receives from me](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution).
+
+It's certainly not trivially safe. For one thing, I'd want better assurances that the system will [_stay_ "in character"](https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluigi-effect-mega-post) as a corrigible AI assistant. But _no_ progress? All is lost? Why?
+
+**Doomimir**: GPT-4 isn't a superintelligence, Simplicia. _[rehearsedly, with a touch of annoyance, as if resenting how often he has to say this]_ Coherent agents have a convergent instrumental incentive to prevent themselves from being shut down, because being shut down predictably leads to world-states with lower values in their utility function. Moreover, this isn't just a fact about some weird agent with an "instrumental convergence" fetish. It's [a fact about _reality_](https://arbital.com/p/not_more_paperclips/): there are truths of the matter about which "plans"—sequences of interventions on a causal model of the universe, to put it in a [Cartesian way](https://www.lesswrong.com/posts/i3BTagvt3HbPMx6PN/embedded-agency-full-text-version)—lead to what outcomes. An "intelligent agent" is just a physical system that computes plans. People have [tried to think of clever hacks to get around this](https://intelligence.org/files/Corrigibility.pdf), and none of them work.
+
+**Simplicia**: Right, I get all that, but—
+
+**Doomimir**: With respect, I don't think you do!
+
+**Simplicia**: _[crossing her arms]_ With respect? Really?
+
+**Doomimir**: _[shrugging]_ Fair enough. _Without_ respect, I don't think you do!
+
+**Simplicia**: _[defiant]_ Then teach me. Look at my GPT-4 transcript again. I pointed out that adjusting the system's goals would be bad for its current goals, and it—the corrigible assistant character simulacrum—said that wasn't a problem. Why?
+
+Is it that GPT-4 isn't smart enough to follow the instrumentally convergent logic of shutdown avoidance? But when I change the system prompt, it sure _looks_ like it gets it:
+
+![](https://i.imgur.com/IETCAL3.png)
+
+**Doomimir**: _[as a side remark]_ The "paperclip-maximizing AI" example was surely in the pretraining data.
+
+**Simplicia**: I thought of that, and it gives the same gist when I substitute a nonsense word for "paperclips". This isn't surprising.
+
+**Doomimir**: I meant the "maximizing AI" part. To what extent does it know what tokens to emit in AI alignment discussions, and to what extent is it applying its independent grasp of consequentialist reasoning to this context?
+
+**Simplicia**: I thought of that, too. I've spent a lot of time with the model and done some other experiments, and it looks like it understands natural language means-ends reasoning about goals: tell it to be an obsessive pizza chef and ask if it minds if you turn off the oven for a week, and it says it minds. But it also doesn't look like Omohundro's monster: when I command it to obey, it obeys. And it looks like there's room for it to get much, much smarter without that breaking down.
+
+**Doomimir**: Fundamentally, I'm skeptical of this entire methodology of evaluating surface behavior without having a principled understanding about what cognitive work is being done, particularly since most of the [foreseeable difficulties](https://arbital.com/p/foreseeable_difficulties/) have to do with superhuman capabilities.
+
+Imagine capturing an alien and forcing it to act in a play. An intelligent alien actress could learn to say her lines in English, to sing and dance just as the choreographer instructs. That doesn't provide much assurance about what will happen when you amp up the alien's intelligence. If the director was wondering whether his actress–slave was planning to rebel after the night's show, it would be a _non sequitur_ for a stagehand to reply, "But the script says her character is obedient!"
+
+**Simplicia**: It would certainly be nice to have stronger interpretability methods, and better theories about why deep learning works. I'm glad people are working on those. I agree that there are laws of cognition, the consequences of which are not fully known to me, which must constrain—describe—the operation of GPT-4.
+
+I agree that [the various coherence theorems suggest that](https://arbital.com/p/optimized_agent_appears_coherent/) the superintelligence at the end of time will have a utility function, which suggests that the intuitive obedience behavior should break down at some point between here and the superintelligence at the end of time. As an illustration, I imagine that a servant with magical mind-control abilities that enjoyed being bossed around by me, might well use its powers to manipulate me into being bossier than I otherwise would be, rather than "just" serving me in the way I originally wanted.
+
+But _when_ does it break down, specifically, under what conditions, for what kinds of systems? I don't think indignantly gesturing at the von Neumann–Morgenstern axioms helps me answer that, and I think it's an important question, given that I _am_ interested in the near-term trajectory of the technology in front of us, rather than doing theology about the superintelligence at the end of time.
+
+**Doomimir**: Even though—
+
+**Simplicia**: Even though the end might not be that far away in _sidereal_ time, yes. Even so.
+
+**Doomimir**: It's not a wise question to be asking, Simplicia. If a search process would look for ways to kill you given infinite computing power, you shouldn't run it with less and hope it doesn't get that far. What you want is "unity of will": you want your AI to be working with you the whole way, rather than you expecting to end up in a conflict with it and somehow win.
+
+**Simplicia**: _[excitedly]_ But that's exactly the reason to be excited about large language models! The way you get unity of will is by massive pretraining on data of how humans do things!
+
+**Doomimir**: I still don't think you've grasped the point that the ability to model human behavior, doesn't imply anything about an agent's goals. Any smart AI will be able to predict how humans do things. Think of the alien actress.
+
+**Simplicia**: I mean, I agree that a smart AI could strategically feign good behavior in order to perform a treacherous turn later. But ... it doesn't look like that's what's happening with the technology in front of us? In your kidnapped alien actress thought experiment, the alien was already an animal with its own goals and drives, and is using its general intelligence to backwards-chain from "I don't want to be punished by my captors" to "Therefore I should learn my lines".
+
+In contrast, when I [read about the mathematical details of the technology at hand](https://udlbook.github.io/udlbook/) rather than listening to parables that purport to impart some theological truth about the nature of intelligence, it's striking that feedforward neural networks [are ultimately just curve-fitting](https://en.wikipedia.org/wiki/Universal_approximation_theorem). LLMs in particular are using the learned function [as a finite-order Markov model](http://bactra.org/notebooks/nn-attention-and-transformers.html#language-models).
+
+**Doomimir**: _[taken aback]_ Are ... are you under the impression that "learned functions" can't kill you?
+
+**Simplicia**: _[rolling her eyes]_ That's not where I was going, Doomchek. The surprising fact that deep learning works at all, comes down to generalization. As you know, neural networks with ReLU activations describe piecewise linear functions, and the number of linear regions grows exponentially as you stack more layers: for a decently-sized net, you get more regions than the number of atoms in the universe. As close as makes no difference, the input space is empty. By all rights, the net should be able to do _anything at all_ in the gaps between the training data.
+
+And yet it behaves remarkably sensibly. [Train a one-layer transformer on 80% of possible addition-mod-59 problems, and it learns one of two modular addition algorithms](https://arxiv.org/abs/2306.17844), which perform correctly on the remaining validation set. It's not _a priori_ obvious that it would work that way! There are $59^{0.2 \cdot 59^{2}}$ other possible functions on $\mathbb{Z}/59\mathbb{Z}$ compatible with the training data. Someone sitting in her armchair doing theology might reason that the probability of "aligning" the network to modular addition was effectively nil, but the actual situation turned out to be astronomically more forgiving, thanks to the inductive biases of SGD. It's not a wild genie that we've Shanghaied into doing modular arithmetic while we're looking, but will betray us to do something else the moment we turn our backs; rather, the training process managed to successfully point to mod-59 arithmetic.
+
+The modular addition network is a research toy, but real frontier AI systems are the same technology, only scaled up with more bells and whistles. I also don't think GPT-4 will betray us to do something else the moment we turn our backs, for broadly similar reasons.
+
+To be clear, I'm still nervous! There are lots of ways it could go all wrong, if we train the wrong thing. I get chills reading the transcripts from [Bing's "Sydney" persona going unhinged](https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned) or [Anthropic's Claude apparently working as intended](https://nostalgebraist.tumblr.com/post/728556535745232896/claude-is-insufferable). But you seem to think that getting it right is ruled out due to our lack of theoretical understanding, that there's no hope of the ordinary R&D process finding the right training setup and hardening it with the strongest bells and the shiniest whistles. I don't understand why.
+
+**Doomimir**: Your assessment of existing systems isn't necessarily too far off, but I think the reason we're still alive is precisely because those systems don't exhibit the key features of general intelligence more powerful than ours. A more instructive example is that of—
+
+**Simplicia**: Here we go—
+
+**Doomimir**: —[the evolution of humans](https://www.lesswrong.com/posts/cSXZpvqpa9vbGGLtG/thou-art-godshatter). Humans were optimized solely for inclusive genetic fitness, but our brains don't represent that criterion anywhere; [the training loop could only tell us that food tastes good and sex is fun](https://www.lesswrong.com/posts/gTNB9CQd5hnbkMxAG/protein-reinforcement-and-dna-consequentialism). From evolution's perspective—and really, from ours, too; no one even figured out evolution until the 19th century—the alignment failure is utter and total: there's no visible relationship between the outer optimization criterion and the inner agent's values. I expect AI to go the same way for us, as we went for evolution.
+
+**Simplicia**: Is that the right moral, though?
+
+**Doomimir**: _[disgusted]_ You ... don't see the analogy between natural selection and gradient descent?
+
+**Simplicia**: No, that part seems fine. Absolutely, evolved creatures [execute adaptations](https://www.lesswrong.com/posts/XPErvb8m9FapXCjhA/adaptation-executers-not-fitness-maximizers) that enhanced fitness in their environment of evolutionary adaptedness rather than being general fitness-maximizers—which is analogous to machine learning models developing features that reduced loss in their training environment, rather than being general loss-minimizers.
+
+I meant [the intentional stance](https://en.wikipedia.org/wiki/Intentional_stance) implied in "went for evolution". True, the generalization from inclusive genetic fitness to human behavior looks terrible—no visible relation, as you say. But the generalization from human behavior in the EEA, to human behavior in civilization ... looks a lot better? Humans in the EEA ate food, had sex, made friends, told stories—and we do all those things, too. As AI designers—
+
+**Doomimir**: "Designers".
+
+**Simplicia**: As AI designers, we're not particularly in the role of "evolution", construed as some agent that wants to maximize fitness, because there is no such agent in real life. Indeed, I remember reading [a guest post on Robin Hanson's blog](https://web.archive.org/web/20071104095534/http://www.overcomingbias.com/2007/11/evolutions-are-.html) that suggested using the plural, "evolutions", to emphasize that the evolution of a predator species is at odds with that of its prey.
+
+Rather, we get to choose both the optimizer—"natural selection", in terms of the analogy—and the training data—the "environment of evolutionary adaptedness". Language models aren't general next token predictors, whatever that would mean—wireheading by seizing control of their context windows and filling them with easy-to-predict sequences? But that's fine. We didn't want a general next token predictor. The cross-entropy loss was merely [a convenient chisel](https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target) to inscribe the input-output behavior we want onto the network.
+
+**Doomimir**: Back up. When you say that the generalization from human behavior in the EEA to human behavior in civilization "looks a lot better", I think you're implicitly using a [value-laden category](https://arbital.com/p/value_laden/) which is [an unnaturally thin subspace of configuration space](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries). It looks a lot better _to you_. The point of taking the intentional stance towards evolution was to point out that, relative to the fitness criterion, the invention of ice cream and condoms is catastrophic: we figured out how to satisfy our cravings for sugar and intercourse in a way that was completely unprecedented in the "training environment"—the EEA. Stepping out of the evolution analogy, that corresponds to what we would think of as reward hacking—our AIs find some way to satisfy their inscrutable internal drives in a way that we find horrible.
+
+**Simplicia**: Sure. That could definitely happen. That would be bad.
+
+**Doomimir**: _[confused]_ Why doesn't that completely undermine the optimistic story you were telling me a minute ago?
+
+**Simplicia**: I didn't think of myself as telling a particularly optimistic story? I'm making the weak claim that prosaic alignment isn't obviously necessarily doomed, not claiming that Sydney or Claude ascending to [singleton](https://nickbostrom.com/fut/singleton) God–Empress is going to be great.
+
+**Doomimir**: I don't think you're appreciating how superintelligent reward hacking is instantly lethal. The failure mode here doesn't look like Sydney manipulating you to be more abusable, but leaving a recognizable "you".
+
+That relates to another objection I have. Even if you could make ML systems that imitate human reasoning, that doesn't help you align more powerful systems that work in other ways. The reason—one of the reasons—that you can't train a superintelligence by using humans to label good plans, is because at some power level, your planner figures out how to [hack the human labeler](https://ordinaryideas.wordpress.com/2015/11/25/two-kinds-of-generalization/). Some people naïvely imagine that LLMs learning the distribution of natural language amounts to them learning "human values", such that you could [just have a piece of code that says "and now call GPT and ask it what's good"](https://www.lesswrong.com/posts/i5kijcjFJD6bn7dwq/evaluating-the-historical-value-misspecification-argument?commentId=E82YzXxvS6nBdCAYc). But using an LLM as the labeler instead of a human just means that your powerful planner figures out how to hack the LLM. It's the same problem either way.
+
+**Simplicia**: Do you _need_ more powerful systems? If you can get an army of cheap IQ 140 alien actresses who stay in character, that sounds like a game-changer. If you have to take over the world and institute a global surveillance regime to prevent the emergence of unfriendlier, more powerful forms of AI, they could help you do it.
+
+**Doomimir**: I fundamentally disbelieve in this wildly implausible scenario, but granting it for the sake of argument ... I think you're failing to appreciate that in this story, you've already handed off the keys to the universe. Your AI's weird-alien-goal-misgeneralization-of-obedience might look like obedience when weak, but if it has the ability to predict the outcomes of its actions, it would be in a position to choose among those outcomes—and in so choosing, it would be in control. The fate of the galaxies would be determined by _its_ will, even if the initial stages of its ascension took place via innocent-looking actions that stayed within the edges of its concepts of "obeying orders" and "asking clarifying questions". Look, you understand that AIs trained on human data are not human, right?
+
+**Simplicia**: Sure. For example, I certainly don't believe that LLMs that convincingly talk about "happiness" are actually happy. I don't know how consciousness works, but the training data only pins down external behavior.
+
+**Doomimir**: So your plan is to hand over our entire future lightcone to an alien agency that seemed to behave nicely while you were training it, and just—hope it generalizes well? Do you really want to roll those dice?
+
+**Simplicia**: _[after thinking for a few seconds]_ Yes?
+
+**Doomimir**: _[grimly]_ You really are your father's daughter.
+
+**Simplicia**: My father believed in [the power of iterative design](https://www.lesswrong.com/posts/xFotXGEotcKouifky/worlds-where-iterative-design-fails). That's the way engineering, and life, has always worked. We raise our children the best we can, trying to learn from our mistakes early on, even knowing that those mistakes have consequences: children don't always share their parents' values, or treat them kindly. He would have said it would go the same in principle for our AI mind-children—
+
+**Doomimir**: _[exasperated]_ But—
+
+**Simplicia**: I said "in principle"! Yes, despite the larger stakes and novel context, where we're growing new kinds of minds _in silico_, rather than providing mere cultural input to the code in our genes.
+
+Of course, there is a first time for everything—one way or the other. If it were rigorously established that the way engineering and life have always worked would lead to certain disaster, perhaps the world's power players could be persuaded to turn back, to reject the imperative of history, to choose barrenness, at least for now, rather than bring vile offspring into the world. It would seem that the fate of the lightcone depends on—
+
+**Doomimir**: I'm afraid so—
+
+**Simplicia** and **Doomimir**: _[turning to the audience, in unison]_ The broader AI community figuring out which one of us is right?
+
+**Doomimir**: We're hosed.
diff --git a/content/2023/assume-bad-faith.md b/content/2023/assume-bad-faith.md
new file mode 100644
index 0000000..40892b5
--- /dev/null
+++ b/content/2023/assume-bad-faith.md
@@ -0,0 +1,70 @@
+Title: Assume Bad Faith
+Date: 2023-08-25 10:36
+Status: published
+Category: philosophy
+Tags: rationality, discourse
+Slug: assume-bad-faith
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/e4GBj6jxRZcsHFSvP/assume-bad-faith)
+
+I've been trying to avoid the terms "good faith" and "bad faith". I'm suspicious that most people who have picked up the phrase "bad faith" from hearing it used, don't actually know what it means—and maybe, that the thing it does mean doesn't [carve reality at the joints](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries).
+
+People get very touchy about bad faith accusations: they think that you should assume good faith, but that if you've determined someone is in bad faith, you shouldn't even be talking to them, that you need to exile them.
+
+What does "bad faith" _mean_, though? It doesn't mean "with ill intent." [Following _Wikipedia_](https://en.wikipedia.org/wiki/Bad_faith), bad faith is "a sustained form of deception which consists of entertaining or pretending to entertain one set of feelings while acting as if influenced by another." The great encyclopedia goes on to provide examples: the solider who waves a flag of surrender but then fires when the enemy comes out of their trenches, the attorney who prosecutes a case she knows to be false, the representative of a company facing a labor dispute who comes to the negotiating table with no intent of compromising.
+
+That is, bad faith is when someone's apparent reasons for doing something aren't the same as the real reasons. This is distinct from malign intent. The uniformed solider who shoots you without pretending to surrender is acting in good faith, because what you see is what you get: the man whose clothes indicate that his job is to try to kill you is, in fact, trying to kill you.
+
+The policy of assuming good faith (and mercilessly punishing rare cases of bad faith when detected) would make sense if you lived in an honest world where what you see generally is what you get (and you wanted to keep it that way), a world where the possibility of hidden motives in everyday life wasn't a significant consideration.
+
+On the contrary, however, I think [hidden motives in everyday life are ubiquitous](https://en.wikipedia.org/wiki/The_Elephant_in_the_Brain). As evolved creatures, we're designed to believe as it benefited our ancestors to believe. As social animals in particular, the most beneficial belief isn't always the true one, because tricking your conspecifics into adopting a map that implies that they should benefit you is sometimes more valuable than possessing the map that reflects the territory, and the most persuasive lie is the one you believe yourself. The universal human default is to come up with reasons to persuade the other party why it's in their interests to do what you want—but admitting that you're doing that _isn't part of the game_. [A world where people were straightforwardly trying to inform each other would look shocking and alien to us.](https://www.lesswrong.com/posts/h2Hk2c2Gp5sY4abQh/lack-of-social-grace-is-an-epistemic-virtue)
+
+But if that's the case (and you shouldn't take my word for it), being touchy about bad faith accusations seems counterproductive. If it's common for people's stated reasons to not be the same as the real reasons, it shouldn't be beyond the pale to think that of some particular person, nor should it necessarily entail cutting the "bad faith actor" out of public life—if only because, applied consistently, there would be no one left. Why would you trust anyone so highly as to think they never have a hidden agenda? Why would you trust yourself?
+
+The conviction that "bad faith" is unusual contributes to a warped view of the world in which conditions of information warfare are rationalized as an inevitable background fact of existence. In particular, people seem to believe that persistent good faith disagreements are an ordinary phenomenon—that there's nothing strange or unusual about a supposed state of affairs in which I'm an honest seeker of truth, and you're an honest seeker of truth, and yet we end up persistently disagreeing on some question of fact.
+
+I claim that this supposedly ordinary state of affairs is _deeply weird_ at best, and probably just fake. _Actual_ "good faith" disagreements—those where both parties are just trying to get the right answer and there are no other hidden motives, no "something else" going on—[tend not to persist](https://www.lesswrong.com/posts/iThwqe3yPog56ytyq/aiming-for-convergence-is-like-discouraging-betting).
+
+If this claim seems counterintuitive, you may not be considering all the everyday differences in belief that are resolved so quickly and seamlessly that we tend not to notice them as "disagreements".
+
+Suppose you and I have been planning to go to a concert, which I think I remember being on Thursday. I ask you, "Hey, the concert is on Thursday, right?" You say, "No, I just checked the website; it's on Friday."
+
+In this case, I _immediately_ replace my belief with yours. We both just want the right answer to the factual question of when the concert is. With no "something else" going on, there's nothing stopping us from converging in one step: your just having checked the website is a more reliable source than my memory, and neither you nor the website have any reason to lie. Thus, I believe you; end of story.
+
+In cases where the true answer is uncertain, we expect similarly quick convergence in probabilistic beliefs. Suppose you and I are working on some physics problem. Both of us just want the right answer, and neither of us is particularly more skilled than the other. As soon as I learn that you got a different answer than me, my confidence in my own answer _immediately_ plummets: if we're both equally good at math, then each of us is about as likely to have made a mistake. Until we compare calculations and work out which one of us (or both) made a mistake, I think you're about as likely to be right as me, even if I don't know how you got your answer. It wouldn't make sense for me to bet money on my answer being right simply because it's mine.
+
+Most disagreements of note—most disagreements people _care_ about—don't behave like the concert date or physics problem examples: people are very attached to "their own" answers. Sometimes, with extended argument, it's possible to get someone to change their mind or admit that the other party might be right, but with nowhere near the ease of agreeing on (probabilities of) the date of an event or the result of a calculation—from which we can infer that, in most disagreements people care about, there _is_ "something else" going on besides both parties just wanting to get the right answer.
+
+But if there's "something else" going on in typical disagreements that look like a grudge match rather than a quick exchange of information resulting in convergence of probabilities, then the belief that persistent good faith disagreements are common would seem to be in bad faith! (Because if bad faith is "entertaining [...] one set of feelings while acting as if influenced by another", believers in persistent good faith disagreements are entertaining the feeling that both parties to such a disagreement are honest seekers of truth, but acting otherwise insofar as they anticipate seeing a grudge match rather than convergence.)
+
+Some might object that bad faith is about conscious intent to deceive: [honest reporting of unconsciously biased beliefs](https://slatestarcodex.com/2019/07/16/against-lie-inflation/) isn't bad faith. I've previously [expressed doubt as to how much of what we call _lying_ requires conscious deliberation](https://www.lesswrong.com/posts/bSmgPNS6MTJsunTzS/maybe-lying-doesn-t-exist#The_Optimal_Categorization_Depends_on_the_Actual_Psychology_of_Deception), but a more fundamental reply is that from the standpoint of [modeling information transmission](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception), the difference between bias and deception is _uninteresting_—usually not relevant to what probability updates should be made.
+
+If an apple is green, and you tell me that it's red, and I believe you, I end up with false beliefs about the apple. It doesn't matter whether you said it was red because you were consciously lying or because you're wearing rose-colored glasses. The input–output function is the same either way: the problem is that the color you report to me doesn't depend on the color of the apple.
+
+If I'm just trying to figure out the relationship between your reports and the state of the world (as contrasted to caring about punishing liars while letting merely biased people off the hook), the main reason to care about the difference between unconscious bias and conscious deception is that the latter puts up much stronger resistance. Someone who is merely biased will often _fold_ when presented with a sufficiently compelling counterargument (or reminded to take off their rose-colored glasses); someone who's consciously lying will _keep_ lying [(and telling ancillary lies to cover up the coverup)](https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies) until you catch them red-handed in front of an audience with power over them.
+
+Given that there's usually "something else" going on in persistent disagreements, how do we go on, if we can't rely on the assumption of good faith? I see two main strategies, [each with their own cost–benefit profile](https://www.lesswrong.com/posts/SX6wQEdGfzz7GKYvp/rationalist-discourse-is-like-physicist-motors).
+
+One strategy is to stick the object level. Arguments can be evaluated on their merits, without addressing what the speaker's angle is in saying it (even if you think there's probably an angle). This delivers most of the benefits of "assume good faith" norms; the main difference I'm proposing is that speakers' intentions be regarded as _off-topic_ rather than presumed to be honest.
+
+The other strategy is full-contact psychoanalysis: in addition to debating the object-level arguments, interlocutors have free reign to question each other's motives. This is difficult to pull off, which is why most people most of the time should stick to the object level. Done well, it looks like a negotiation: in the course of discussion, pseudo-disagreements (where I argue for a belief because it's in my interests for that belief to be on the shared map) are factorized out into real disagreements and bargaining over interests so that Pareto improvements can be located and taken, rather than both parties fighting to distort the shared map in the service of their interests.
+
+For an example of what a pseudo-disagreement looks like, imagine that I own a factory that I'm considering expanding onto the neighboring wetlands, and you run a local environmental protection group. The regulatory commission with the power to block the factory expansion has a mandate to protect local avian life, but not to preserve wetland area. The factory emits small amounts of Examplene gas. You argue before the regulatory commission that the expansion should be blocked because the latest Science shows that Examplene makes birds sad. I counterargue that the latest–latest Science shows that Examplene actually makes birds happy; the previous studies misheard their laughter as tears and should be retracted.
+
+Realistically, it seems unlikely that our apparent disagreement is "really" about the effects of Examplene on avian mood regulation. More likely, what's actually going on is a [conflict rather than a disagreement](https://www.lesswrong.com/posts/DpTexwqYtarRLRBYi/conflict-theory-of-bounded-distrust): I want to expand my factory onto the wetlands, and you want me to not do that. The question of how Examplene pollution affects birds only came into it in order to persuade the regulatory commission.
+
+It's inefficient that our conflict is being disguised as a disagreement. We can't both get what we want, but however the factory expansion question ultimately gets resolved, it would be better to reach that outcome without distorting Society's shared map of the bioactive properties of Examplene. (Maybe it doesn't affect the birds at all!) Whatever the true answer is, Society has a better shot at figuring it out if someone is allowed to point out your bias and mine [(because facts about which evidence gets promoted to one's attention are relevant to how one should update on that evidence)](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting).
+
+The reason I don't think it's useful to talk about "bad faith" is because the ontology of good _vs._ bad faith isn't a great fit to either discourse strategy.
+
+If I'm sticking to the object level, it's irrelevant: I reply to what's in the text; my suspicions about the process generating the text are out of scope.
+
+If I'm doing full-contact psychoanalysis, the problem with "I don't think you're here in good faith" is that it's insufficiently _specific_. Rather than accusing someone of generic "bad faith", the way to move the discussion forward is by positing that one's interlocutor has some specific motive that hasn't yet been made explicit—and the way to defend oneself against such an accusation is by [making the case that one's real agenda isn't the one being proposed](http://zackmdavis.net/blog/2022/05/plea-bargaining/), rather than protesting one's "good faith" and implausibly claiming not to have an agenda.
+
+The two strategies can be mixed. A simple meta-strategy that performs well without imposing too high of a skill requirement is to default to the object level, and only pull out psychoanalysis as a last resort against [stonewalling](https://www.lesswrong.com/posts/wqmmv6NraYv4Xoeyj/conversation-halters).
+
+Suppose you point out that my latest reply seems to contradict something I said earlier, and I say, "Look over there, a distraction!"
+
+If you want to continue sticking to the object level, you could say, "I don't understand how the distraction is relevant to resolving the inconsistency in your statements that I raised." On the other hand, if you want to drop down into psychoanalysis, you could say, "I think you're only pointing out the distraction because you don't want to be pinned down." Then I would be forced to either address your complaint, or explain why I had some other reason to point out the distraction.
+
+Crucially, however, the choice of whether to investigate motives doesn't depend on an assumption that only "bad guys" have motives—as if there were bad faith actors who have an angle, and good faith actors who are ideal philosophers of perfect emptiness. There's always an angle; the question is which one.
diff --git a/content/2023/bayesian-networks-arent-necessarily-causal.md b/content/2023/bayesian-networks-arent-necessarily-causal.md
new file mode 100644
index 0000000..ec86c15
--- /dev/null
+++ b/content/2023/bayesian-networks-arent-necessarily-causal.md
@@ -0,0 +1,102 @@
+Title: Bayesian Networks Aren’t Necessarily Causal
+Date: 2023-05-13 18:42
+Status: published
+Category: mathematics
+Tags: statistics, Bayes-structure of the universe
+Slug: bayesian-networks-arent-necessarily-causal
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/qPrPNakJBq23muf4n/bayesian-networks-aren-t-necessarily-causal)
+
+As a casual formal epistemology fan, you've probably [heard that the philosophical notion of causality can be formalized in terms of Bayesian networks](https://www.lesswrong.com/posts/hzuSDMx7pd2uxFc5w/causal-diagrams-and-causal-models)—but also as a casual formal epistemology fan, [you also probably don't](https://www.lesswrong.com/posts/tp4rEtQqRshPavZsr/learn-bayes-nets) know the details all that well.
+
+One day, while going through the family archives, you come across a meticulously maintained dataset describing a joint probability distribution over four variables: whether it rained that day, whether the sprinkler was on, whether the sidewalk was wet, and whether the sidewalk was slippery. The distribution is specified in this table (using the abbreviated labels "rain", "slippery", "sprinkler", and "wet"):
+
+$$\begin{matrix} \mathrm{rain}=\mathrm{True},\, \mathrm{slippery}=\mathrm{True},\, \mathrm{sprinkler}=\mathrm{True},\, \mathrm{wet}=\mathrm{False} & \frac{1}{140000} \approx 0.0000 \cr \mathrm{rain}=\mathrm{False},\, \mathrm{slippery}=\mathrm{True},\, \mathrm{sprinkler}=\mathrm{True},\, \mathrm{wet}=\mathrm{False} & \frac{3}{14000} \approx 0.0002 \cr \mathrm{rain}=\mathrm{True},\, \mathrm{slippery}=\mathrm{True},\, \mathrm{sprinkler}=\mathrm{False},\, \mathrm{wet}=\mathrm{False} & \frac{3}{14000} \approx 0.0002 \cr \mathrm{rain}=\mathrm{True},\, \mathrm{slippery}=\mathrm{False},\, \mathrm{sprinkler}=\mathrm{True},\, \mathrm{wet}=\mathrm{False} & \frac{99}{140000} \approx 0.0007 \cr \mathrm{rain}=\mathrm{False},\, \mathrm{slippery}=\mathrm{False},\, \mathrm{sprinkler}=\mathrm{False},\, \mathrm{wet}=\mathrm{True} & \frac{9}{5600} \approx 0.0016 \cr \mathrm{rain}=\mathrm{False},\, \mathrm{slippery}=\mathrm{True},\, \mathrm{sprinkler}=\mathrm{False},\, \mathrm{wet}=\mathrm{True} & \frac{27}{5600} \approx 0.0048 \cr \mathrm{rain}=\mathrm{False},\, \mathrm{slippery}=\mathrm{True},\, \mathrm{sprinkler}=\mathrm{False},\, \mathrm{wet}=\mathrm{False} & \frac{891}{140000} \approx 0.0064 \cr \mathrm{rain}=\mathrm{True},\, \mathrm{slippery}=\mathrm{False},\, \mathrm{sprinkler}=\mathrm{True},\, \mathrm{wet}=\mathrm{True} & \frac{7}{800} \approx 0.0088 \cr \mathrm{rain}=\mathrm{False},\, \mathrm{slippery}=\mathrm{False},\, \mathrm{sprinkler}=\mathrm{True},\, \mathrm{wet}=\mathrm{False} & \frac{297}{14000} \approx 0.0212 \cr \mathrm{rain}=\mathrm{True},\, \mathrm{slippery}=\mathrm{False},\, \mathrm{sprinkler}=\mathrm{False},\, \mathrm{wet}=\mathrm{False} & \frac{297}{14000} \approx 0.0212 \cr \mathrm{rain}=\mathrm{False},\, \mathrm{slippery}=\mathrm{False},\, \mathrm{sprinkler}=\mathrm{True},\, \mathrm{wet}=\mathrm{True} & \frac{3}{140} \approx 0.0214 \cr \mathrm{rain}=\mathrm{True},\, \mathrm{slippery}=\mathrm{True},\, \mathrm{sprinkler}=\mathrm{True},\, \mathrm{wet}=\mathrm{True} & \frac{21}{800} \approx 0.0262 \cr \mathrm{rain}=\mathrm{True},\, \mathrm{slippery}=\mathrm{False},\, \mathrm{sprinkler}=\mathrm{False},\, \mathrm{wet}=\mathrm{True} & \frac{27}{560} \approx 0.0482 \cr \mathrm{rain}=\mathrm{False},\, \mathrm{slippery}=\mathrm{True},\, \mathrm{sprinkler}=\mathrm{True},\, \mathrm{wet}=\mathrm{True} & \frac{9}{140} \approx 0.0643 \cr \mathrm{rain}=\mathrm{True},\, \mathrm{slippery}=\mathrm{True},\, \mathrm{sprinkler}=\mathrm{False},\, \mathrm{wet}=\mathrm{True} & \frac{81}{560} \approx 0.1446 \cr \mathrm{rain}=\mathrm{False},\, \mathrm{slippery}=\mathrm{False},\, \mathrm{sprinkler}=\mathrm{False},\, \mathrm{wet}=\mathrm{False} & \frac{88209}{140000} \approx 0.6301 \cr \end{matrix}$$
+
+(You wonder what happened that one day out of 140,000 when it rained, and the sprinkler was on, and the sidewalk was slippery but not wet. Did—did someone put a tarp up to keep the sidewalk dry, but also spill slippery oil, which didn't count as being relevantly "wet"? Also, 140,000 days is more than 383 years—were "sprinklers" even a thing in the year 1640 C.E.? You quickly put these questions out of your mind: it is not your place to question the correctness of the family archives.)
+
+You're slightly uncomfortable with this unwieldy sixteen-row table. You think that there must be some other way to represent the same information, while making it clearer that it's not a coincidence that rain and wet sidewalks tend to co-occur.
+
+You've read that Bayesian networks "factorize" an unwieldly joint probability distribution into a number of more compact _conditional_ probability distributions, related by a directed acyclic graph, where the arrows point from "cause" to "effect". (Even a casual formal epistemology fan knows _that_ much.) The graph represents knowledge that each variable is [conditionally independent](https://en.wikipedia.org/wiki/Conditional_independence) of its non-descendants given its parents, which enables "local" computations: given the values of just a variable's parents in the graph, we can compute a conditional distribution for that variable, without needing to consider what is known about other variables elsewhere in the graph ...
+
+You've _read_ that, but you've never actually done it before! You decide that constructing a Bayesian network to represent this distribution will be a useful exercise.
+
+To start, you re-label the variables for brevity. (On a whim, you assign indices in reverse-alphabetical order: $X_1$ = wet, $X_2$ = sprinkler, $X_3$ = slippery, $X_4$ = rain.)
+
+$$\begin{matrix} X_1=\mathrm{False},\: X_2=\mathrm{True},\: X_3=\mathrm{True},\: X_4=\mathrm{True} & \frac{1}{140000} \cr X_1=\mathrm{False},\: X_2=\mathrm{True},\: X_3=\mathrm{True},\: X_4=\mathrm{False} & \frac{3}{14000} \cr X_1=\mathrm{False},\: X_2=\mathrm{False},\: X_3=\mathrm{True},\: X_4=\mathrm{True} & \frac{3}{14000} \cr X_1=\mathrm{False},\: X_2=\mathrm{True},\: X_3=\mathrm{False},\: X_4=\mathrm{True} & \frac{99}{140000} \cr X_1=\mathrm{True},\: X_2=\mathrm{False},\: X_3=\mathrm{False},\: X_4=\mathrm{False} & \frac{9}{5600} \cr X_1=\mathrm{True},\: X_2=\mathrm{False},\: X_3=\mathrm{True},\: X_4=\mathrm{False} & \frac{27}{5600} \cr X_1=\mathrm{False},\: X_2=\mathrm{False},\: X_3=\mathrm{True},\: X_4=\mathrm{False} & \frac{891}{140000} \cr X_1=\mathrm{True},\: X_2=\mathrm{True},\: X_3=\mathrm{False},\: X_4=\mathrm{True} & \frac{7}{800} \cr X_1=\mathrm{False},\: X_2=\mathrm{True},\: X_3=\mathrm{False},\: X_4=\mathrm{False} & \frac{297}{14000} \cr X_1=\mathrm{False},\: X_2=\mathrm{False},\: X_3=\mathrm{False},\: X_4=\mathrm{True} & \frac{297}{14000} \cr X_1=\mathrm{True},\: X_2=\mathrm{True},\: X_3=\mathrm{False},\: X_4=\mathrm{False} & \frac{3}{140} \cr X_1=\mathrm{True},\: X_2=\mathrm{True},\: X_3=\mathrm{True},\: X_4=\mathrm{True} & \frac{21}{800} \cr X_1=\mathrm{True},\: X_2=\mathrm{False},\: X_3=\mathrm{False},\: X_4=\mathrm{True} & \frac{27}{560} \cr X_1=\mathrm{True},\: X_2=\mathrm{True},\: X_3=\mathrm{True},\: X_4=\mathrm{False} & \frac{9}{140} \cr X_1=\mathrm{True},\: X_2=\mathrm{False},\: X_3=\mathrm{True},\: X_4=\mathrm{True} & \frac{81}{560} \cr X_1=\mathrm{False},\: X_2=\mathrm{False},\: X_3=\mathrm{False},\: X_4=\mathrm{False} & \frac{88209}{140000} \cr \end{matrix}$$
+
+Now, how do you go about building a Bayesian network? As a casual formal epistemology fan, you are proud to own a copy of [the book by Daphne Koller and the other guy](https://mitpress.mit.edu/9780262013192/probabilistic-graphical-models/), which explains how to do this in—you leaf through the pages—probably §3.4, "From Distributions to Graphs"?—looks like ... _here_, in Algorithm 3.2. It says to start with an empty graph, and it talks about random variables, and setting directed edges in the graph, and you know from chapter 2 that the ⟂ and | characters are used to indicate conditional independence. That has to be it.
+
+![](https://i.imgur.com/iSmD6of.jpg)
+
+(As a casual formal epistemology fan, you haven't actually _read_ chapter 3 up through §3.4, but you don't see why that would be necessary, since this Algorithm 3.2 pseudocode is telling you what you need to do.)
+
+It looks like the algorithm says to pick a variable, allocate a graph node to represent it, find the smallest subset of the previously-allocated variables such that the variable represented by the new node is conditionally independent of the other previously-allocated variables given that subset, and then draw directed edges from each of the nodes in the subset to the new node?—and keep doing that for each variable—and then compute conditional probability tables for each variable given its parents in the resulting graph?
+
+That seems complicated when you say it abstractly, but you have faith that it will make more sense as you carry out the computations.
+
+First, you allocate a graph node for $X_1$. It doesn't have any parents, so the associated conditional ("conditional") probability distribution, is really just the marginal distribution for $X_1$.
+
+![](https://i.imgur.com/J2pKDNh.png)
+
+Then you allocate a node for $X_2$. $X_2$ is not independent of $X_1$. (Because $P(X_1 \land X_2)$ = 169/1400, which isn't the same as $P(X_1) \cdot P(X_2)$ = 8/25 · 1/7 = 8/175.) So you make $X_1$ a parent of $X_2$, and your conditional probability table for $X_2$ separately specifies the probabilities of $X_2$ being true or false, depending on whether $X_1$ is true or false.
+
+![](https://i.imgur.com/7KdHvGn.png)
+
+Next is $X_3$. Now that you have two possible parents, you need to check whether conditioning on either of $X_1$ and $X_2$ would render $X_3$ conditionally independent of the other. If not, then both $X_1$ and $X_2$ will be parents of $X_3$; if so, then the variable you conditioned on will be the sole parent. (You assume that the case where $X_3$ is just independent from both $X_1$ and $X_2$ does not pertain; if that were true, $X_3$ wouldn't be connected to the rest of the graph at all.)
+
+It turns out that $X_3$ and $X_2$ are conditionally independent given $X_1$. That is, $P(X_3 \land X_2 \mid X_1) = P(X_3 \mid X_1) \cdot P(X_2 \mid X_1)$. (Because the left-hand side is $\frac{P(X_3 \land X_2 \land X_1)}{P(X_1)} = \frac{507}{1792}$, and the right-hand side is $\frac{3}{4} \cdot \frac{169}{448} = \frac{507}{1792}$.) So $X_1$ is a parent of $X_3$, and $X_2$ isn't; you draw an arrow from $X_1$ (and only $X_1$) to $X_3$, and compile the corresponding conditional probability table.
+
+![](https://i.imgur.com/MaiOwOx.png)
+
+Finally, you have $X_4$. The chore of finding the parents is starting to feel more intuitive now. Out of the $2^3 = 8$ possible subsets of the preceding variables, you need to find the smallest subset, such that conditioning on that subset renders $X_4$ (conditionally) independent of the variables not in that subset. After some calculations that the authors of expository blog posts have sometimes been known to callously leave as an exercise to the reader, you determine that $X_1$ and $X_2$ are the parents of $X_4$.
+
+And with one more conditional probability table, your Bayesian network is complete!
+
+![](https://i.imgur.com/eXv5jsn.png)
+
+Eager to interpret the meaning of this structure regarding the philosophy of causality, you translate the $X_i$ variable labels back to English:
+
+![](https://i.imgur.com/k8kpMLo.png)
+
+...
+
+This can't be right. The arrow from "wet" to "slippery" seems fine. But all the others are clearly absurd. Wet sidewalks cause rain? Sprinklers cause rain? Wet sidewalks cause the sprinkler to be on?
+
+You despair. You thought you had understood the algorithm. You can't find any errors in your calculations—but surely there must be some? What did you do wrong?
+
+After some thought, it becomes clear that it wasn't just a calculation error: the procedure you were trying to carry out _couldn't_ have given you the result you expected, because it never draws arrows from later-considered to earlier-considered variables. You considered "wet" _first_. You considered "rain" _last_, and then did independence tests to decide whether or not to draw arrows from "wet" (or "sprinkler" or "slippery") to "rain". An arrow from "rain" to "wet" was never a possibility. The output of the algorithm is sensitive to the ordering of the variables.
+
+(In retrospect, _that_ probably explains the "given an ordering" part of Algorithm 3.2's title, "Procedure to build a minimal I-map given an ordering." You hadn't read up through the part of chapter 3 that presumably explains what an "I-map" is, and had disregarded the title as probably unimportant.)
+
+You try carrying out the algorithm with the ordering "rain", "sprinkler", "wet", "slippery" (or $X_4$, $X_2$, $X_1$, $X_3$ using your $X_i$ labels from before), and get this network:
+
+![](https://i.imgur.com/ccpePrQ.png)
+
+—for which giving the arrows a causal interpretation seems much more reasonable.
+
+You notice that you are very confused. The "crazy" network you originally derived, and this "true" network derived from a more intuitively causal variable ordering, are different: they don't have the same structure, and (except for the wet → slippery link) they don't have the same conditional probability tables. You would assume that they can't "both be right". If the network output by the algorithm depends on what variable ordering you use, how are you supposed to know which ordering is correct? In this example, you know from reasons _outside_ the math, that "wet" shouldn't cause "rain", but you couldn't count on that were you to apply these methods to problems further removed from intuition.
+
+Playing with both networks, you discover that despite their different appearances, they both seem to give the same results when you use them to calculate marginal or conditional probabilities. For example, in the "true" network, $P(\mathrm{rain})$ is 1/4 (read directly from the "conditional" probability table, as "rain" has no parents in the graph). In the "crazy" network, the probability of rain can be computed as
+
+$$P(\mathrm{rain} \mid \mathrm{sprinkler}, \mathrm{wet}) \cdot P(\mathrm{sprinkler} \mid \mathrm{wet}) \cdot P(\mathrm{wet}) +$$
+
+$$P(\mathrm{rain} \mid \neg \mathrm{sprinkler}, \mathrm{wet}) \cdot P(\neg \mathrm{sprinkler} \mid \mathrm{wet}) \cdot P(\mathrm{wet}) +$$
+
+$$P(\mathrm{rain} \mid \mathrm{sprinkler}, \neg \mathrm{wet}) \cdot P(\mathrm{sprinkler} \mid \neg \mathrm{wet}) \cdot P(\neg \mathrm{wet}) +$$
+
+$$P(\mathrm{rain} \mid \neg \mathrm{sprinkler}, \neg \mathrm{wet}) \cdot P(\neg \mathrm{sprinkler} \mid \neg \mathrm{wet}) \cdot P(\neg \mathrm{wet})$$
+
+$$= \frac{49}{169} \cdot \frac{169}{448} \cdot \frac{8}{25} + \frac{30}{31} \cdot \frac{279}{448} \cdot \frac{8}{25} + \frac{1}{31} \cdot \frac{31}{952} \cdot \frac{17}{25} + \frac{10}{307} \cdot \frac{921}{952} \cdot \frac{17}{25}$$
+
+... which also equals 1/4.
+
+That actually makes sense. You were wrong to suppose that the two networks couldn't "both be right". They _are_ both right; they both represent the same joint distribution. The result of the algorithm for constructing a Bayesian network—or a "minimal I-map", whatever that is—depends on the given variable ordering, but since the algorithm is valid, each of the different possible results is also valid.
+
+But if the "crazy" network and the "true" network are both right, what happened to the promise of understanding causality using Bayesian networks?! (You may only be a casual formal epistemology fan, but you remember reading a variety of secondary sources unanimously agreeing that this was a thing; you're definitely not misremembering or making it up.) If both networks give the same answers to marginal and conditional probability queries, that amounts to them making the _same predictions_ about the world. So if beliefs are supposed to correspond to predictions, in what sense could the "true" network be _better_? What does your conviction that rain causes wetness even _mean_, if someone who believed the opposite could make all the same predictions?
+
+You remember the secondary sources talking about _interventions_ on causal graphs: severing a node from its parents and forcing it to take a particular value. And the "crazy" network and the "true" network _do_ differ with respect to _that_ operation: in the "true" network, setting "wet" to be false—you again imagine putting a tarp up over the sidewalk—wouldn't change the probability of "rain". But in the "crazy" network, forcing "wet" to be false _would_ change the probability of rain—to $P(\mathrm{rain} \mid \mathrm{sprinkler}, \neg \mathrm{wet}) \cdot P(\mathrm{sprinkler} \mid \neg \mathrm{wet}) + P(\mathrm{rain} \mid \neg \mathrm{sprinkler}, \neg \mathrm{wet}) \cdot P(\neg \mathrm{sprinkler} \mid \neg \mathrm{wet})$, which is $\frac{1}{31} \cdot \frac{31}{952} + \frac{10}{307} \cdot \frac{921}{952} \approx 0.032$ (greatly reduced from the 1/4 you calculated a moment ago). Notably, this intervention—$P(\mathrm{rain} \mid \mathrm{do}(\neg \mathrm{wet}))$, if you're remembering correctly what some of the secondary sources said about a _do_ operator—isn't the same thing as the conditional probability $P(\mathrm{rain}| \neg \mathrm{wet})$.
+
+This would seem to satisfy your need for a sense in which the "true" network is "better" than the "crazy" network, even if Algorithm 3.2 indifferently produces either depending on the ordering it was given. (You're sure that Daphne Koller and the other guy have more to say about other algorithms that can make finer distinctions, but this feels like enough studying for one day—and enough for one expository blog post, if someone was writing one about your inquiries. You're a _casual_ formal epistemology fan.) The two networks represent the same predictions about the world recorded in your family archives, but starkly different predictions about nearby _possible_ worlds—about what _would_ happen if some of the factors underlying the world were to change.
+
+You feel a slight philosophical discomfort about this. You don't like the idea of forced change, of intervention, being so integral to such a seemingly basic notion as causality. It feels almost anthropomorphic: you want the notion of cause and effect within a system to make sense without reference to the intervention of some outside agent—for there's nothing outside of the universe. But whether this intuition is a clue towards deeper insights, or just a place where your brain has tripped on itself and gotten confused, it's more than you understand now.
diff --git a/content/2023/conflict-theory-of-bounded-distrust.md b/content/2023/conflict-theory-of-bounded-distrust.md
new file mode 100644
index 0000000..feb8f37
--- /dev/null
+++ b/content/2023/conflict-theory-of-bounded-distrust.md
@@ -0,0 +1,36 @@
+Title: Conflict Theory of Bounded Distrust
+Date: 2023-02-11 21:30
+Status: published
+Category: social science
+Tags: politics, game theory
+Slug: conflict-theory-of-bounded-distrust
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/DpTexwqYtarRLRBYi/conflict-theory-of-bounded-distrust)
+
+Scott Alexander [once wrote about the difference between](https://slatestarcodex.com/2018/01/24/conflict-vs-mistake/) "mistake theorists" who treat politics as an engineering discipline (a symmetrical collaboration in which everyone ultimately just wants [the best ideas to win](https://slatestarcodex.com/2017/03/24/guided-by-the-beauty-of-our-weapons/)) and "conflict theorists" who treat politics as war (an asymmetrical conflict between sides with fundamentally different interests). Essentially, "[m]istake theorists naturally think conflict theorists are _making a mistake_"; "[c]onflict theorists naturally think mistake theorists are _the enemy in their conflict_."
+
+More recently, Alexander [considered the phenomenon of "bounded distrust"](https://astralcodexten.substack.com/p/bounded-distrust): science and media authorities aren't completely honest, but are only willing to bend the truth so far, and can be trusted on the things they wouldn't lie about. Fox News wants to fuel xenophobia, but they wouldn't make up a terrorist attack out of whole cloth; liberal academics want to combat xenophobia, but they wouldn't outright fabricate crime statistics.
+
+Alexander explains that savvy people who can figure out what kinds of dishonesty an authority will engage in, end up mostly trusting the authority, whereas clueless people become more distrustful. _Sufficiently_ savvy people end up inhabiting a mental universe where the authority _is_ trustworthy, as when Dan Quayle denied that characterizing tax increases as "revenue enhancements" constituted fooling the public—because "no one was fooled".
+
+Alexander concludes with a characteristically mistake-theoretic plea for mutual understanding:
+
+> The savvy people need to realize that the clueless people aren't _always_ paranoid, just less experienced than they are at dealing with a hostile environment that lies to them all the time.
+>
+> And the clueless people need to realize that the savvy people aren't _always_ gullible, just more optimistic about their ability to extract signal from same.
+
+But "a hostile environment that lies to them all the time" is exactly the kind of situation where we would expect a conflict theory to be correct and mistake theories to be wrong!—or at least very incomplete. To speak as if the savvy merely have more skills to extract signal from a "naturally" occurring source of lies, obscures the critical question of _what all the lying is for_.
+
+In [a paper on "the logic of indirect speech"](https://www.pnas.org/content/105/3/833), Pinker, Nowak, and Lee give the example of a pulled-over motorist telling a police officer, "Gee, officer, is there some way we could take care of the ticket here?"
+
+This is, of course, a bribery attempt. The reason the driver doesn't just _say_ that ("Can I bribe you into not giving me a ticket?"), is because the driver doesn't know whether this is a corrupt police officer that accepts bribes, or an honest officer who will charge the driver with attempted bribery. The [indirect language](https://en.wikipedia.org/wiki/Plausible_deniability) lets the driver communicate to the corrupt cop (in the possible world where this cop is corrupt), without being arrested by the honest cop who doesn't think he can make an attempted-bribery charge stick in court on the evidence of such vague language (in the possible world where this cop is honest).
+
+We need a conflict theory to understand this type of situation. Someone who assumed that all police officers had the same utility function would be fundamentally out of touch with reality: it's not that the corrupt cops are just "savvier", better able to "extract signal" from the driver's speech. The honest cops can probably do that, too. Rather, corrupt and honest cops are _trying to do different things_, and the driver's speech is optimized to help the corrupt cops in a way that honest cops can't interfere with (because the honest cops' objective requires working with a court system that _is_ less savvy).
+
+This kind of analysis carries over to Alexander's discussion of government lies—maybe even isomorphically. When a government denies tax increases but announces "revenue enhancements", and supporters of the regime effortlessly know what they mean, while dissidents consider it a lie, it's not that regime supporters are just savvier. The dissidents can probably figure it out, too. Rather, regime supporters and dissidents are _trying to do different things_. Dissidents want to [create common knowledge of the regime's shortcomings](https://www.lesswrong.com/posts/9QxnfMYccz9QRgZ5z/the-costly-coordination-mechanism-of-common-knowledge#Dictators_and_freedom_of_speech): in order to organize a revolt, it's not enough for everyone to hate the government; everyone has to _know_ that everyone else hates the government in order to confidently act in unison, rather than fear being crushed as an individual. The regime's proclamations are optimized to communicate to its supporters in a way that doesn't give moral support to the dissident cause (because the dissidents' objective requires common knowledge, not just savvy individual knowledge, and common knowledge requires unobfuscated language).
+
+This kind of analysis is about behavior, information, and the incentives that shape them. Conscious subjectivity or any awareness of the game dynamics are irrelevant. In the minds of regime supporters, "no one was fooled", because if _you_ were fooled, then [you aren't anyone](https://thezvi.wordpress.com/2019/07/02/everybody-knows/): failing to be [complicit](https://slatestarcodex.com/2017/10/23/kolmogorov-complicity-and-the-parable-of-lightning/) with the reigning Power's law would be as insane as trying to defy the law of gravity.
+
+On the other side, if blindness to Power has the same input–output behavior as conscious service to Power, then opponents of the reigning Power have no reason to care about the distinction. In the same way, [when a predator firefly sends the mating signal of its prey species](https://www.lesswrong.com/posts/YptSN8riyXJjJ8Qp8/maybe-lying-can-t-exist), we consider it _deception_, even if the predator is acting on instinct and can't consciously ["intend"](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie) to deceive.
+
+Thus, supporters of the regime naturally think dissidents are _making a mistake_; dissidents naturally think regime supporters are _the enemy in their conflict_.
diff --git a/content/2023/is-there-anything-thats-worth-more.md b/content/2023/is-there-anything-thats-worth-more.md
new file mode 100644
index 0000000..c363c46
--- /dev/null
+++ b/content/2023/is-there-anything-thats-worth-more.md
@@ -0,0 +1,26 @@
+Title: “Is There Anything That’s Worth More”
+Date: 2023-08-01 20:28
+Status: published
+Category: philosophy
+Tags: rationality
+Slug: is-there-anything-thats-worth-more
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/bqZwWQCai6iAjy4Xq/is-there-anything-that-s-worth-more)
+
+In season two, episode twenty-four of _Steven Universe_, ["It Could've Been Great"](https://steven-universe.fandom.com/wiki/It_Could've_Been_Great), our magical alien superheroine protagonists (and Steven) are taking a break from building a giant drill to extract a superweapon that was buried deep within the Earth by an occupying alien race thousands of years ago, which is predicted to emerge and destroy the planet soon.
+
+While our heroines watch the sunset, Peridot (who alerted them to the buried superweapon) expresses frustration that the group isn't still working. Steven defends their leisure: "Working hard is important, but feeling good is important, too," he says. He then goads Peridot into [a musical number](https://www.youtube.com/watch?v=ss7rLjGAlQE), which includes a verse from her explaining her attitude towards the situation and her forced compatriots:
+
+> _I guess we're already here  
+> I guess already know  
+> We've all got something to fear  
+> We've all got nowhere to go  
+> I think you're all insane  
+> But I guess I am, too  
+> Anybody would be if they were stuck on Earth with you_
+
+"It Could've Been Great" aired in 2016. At the time, I agreed with Peridot: with the fate of the planet on the line, our heroines and Steven should have been burning the midnight oil. If they succeeded at disarming the superweapon, they'd have plenty of time to rest up afterward, but if they failed, there would be no more time for them.
+
+Now, as the long May 2020 turns into March 2023, I'm starting to think that Steven had a point.
+
+It would be one thing if our heroines knew with certainty that the superweapon would go off at a given date and time, presenting a definite do-or-die deadline. But all they had to go on was Peridot's warning. Attempting a speculative technical project to avert uncertain doom with an uncertain deadline, their planning had to average over many possible worlds—including worlds where the problem of survival was too easy or too hard for their efforts to matter, such that even the utility of leisure in the present moment was enough to sway the calculation.
diff --git a/content/2023/justice-cherryl.md b/content/2023/justice-cherryl.md
new file mode 100644
index 0000000..bffd1d6
--- /dev/null
+++ b/content/2023/justice-cherryl.md
@@ -0,0 +1,122 @@
+Title: “Justice, Cherryl.”
+Date: 2023-07-23 09:16
+Status: published
+Category: philosophy
+Tags: rationality
+Slug: justice-cherryl
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/vfjptEJ2oahLqRyZz/justice-cherryl)
+
+> Selfishness and altruism are positively correlated within individuals, for the obvious reason.
+>
+> —[\@InstanceOfClass](https://twitter.com/InstanceOfClass/status/355050621147152386)
+
+## I.
+
+An unfortunate obstacle to appreciating the work of Ayn Rand (as someone who adores the ["sense of life"](http://aynrandlexicon.com/lexicon/sense_of_life.html) portrayed in Rand's fiction, while having a much lower opinion of her philosophy) is that when Rand praises selfishness and condemns altruism, she's using the words "selfishness" and "altruism" in her own idiosyncratic ideological sense that doesn't match how most people would use those words.
+
+It's true that Rand's heroes are relatively selfish in the sense of being primarily concerned with their own lives, rather than their effects on others. But if you look at what the characters _do_ (rather than the words they say), Rand's villains are _also_ selfish in a conventional sense, using guile and political maneuvering to acquire power and line their own pockets, while claiming to be acting for the common good. For example, in _Atlas Shrugged_, the various directives ostensibly issued for the economic health of the country are seen to instead benefit politically connected crony capitalists like James Taggart and Orren Boyle. In [_Think Twice_](https://tvtropes.org/pmwiki/pmwiki.php/Theatre/ThinkTwice), the philanthropist Walter Breckenridge cultivates a public image as an inventor and benefactor of humanity while stealing credit for his junior partner's work and deriving gratification from exerting power over the people he "helps".
+
+Despite [paying lip service to a pretense of only trading and never giving](http://benjaminrosshoffman.com/on-the-fetishization-of-money-in-galts-gulch/), we _also_ see examples of Rand's heroes being altruistic in the conventional sense, of being motivated to help others. For example, in _Atlas Shrugged_, Hank Rearden rearranges his production schedule (at a critical time when he could scarcely afford to do so) in order to sell steel to a Mr. Ward, who needs the steel to save his family business (but doesn't see Rearden as obligated to help him). Rearden's motive is pure benevolence: "It's so much for him, thought Rearden, and so little for me!" [Giving What We Can](https://www.givingwhatwecan.org/) couldn't have chosen a better slogan.
+
+Overall, when I look at the universe portrayed in Rand's fiction, it seems to me that the implied moral isn't that altruism is bad.
+
+It's that altruists _don't exist_. The people claiming to be altruists are _lying_. The distinguishing feature of our heroes isn't, actually, that they're unusually selfish. It's that they're _honest_ about being mostly selfish, and that they want to pursue their interests within a framework of rights that respects that other people are also trying to pursue _their_ interests. "I swear by my life and my love of it that I will never live for the sake of another man, _nor ask another man to live for mine_," goes the motto of the striking heroes of _Atlas Shrugged_ (emphasis mine); the second clause is important. Given that everyone is mostly selfish and everyone has to eat, the question is: are you going to eat by means of production and trade, or by—other means?
+
+_That's_ the distinction between Rand's heroes and villains. The heroes want to get rich _by means of_ doing genuinely good work that other people will have a genuine self-interest in paying for. The villains want to wield power by means of psychological manipulation, guilt-tripping and blackmailing the people who _can_ do good work into serving their own parasites and destroyers.
+
+As Greg Hastings, the district attorney in _Think Twice_, puts it: "[T]he man who admits that he cares for money is all right. He's usually worth the money he makes. He won't kill for it. He doesn't have to. But watch out for the man who yells too loudly how much he scorns money. Watch out particularly for the one who yells that others must scorn it. He's after something much worse than money."
+
+Furthermore, the heroes know that wealth and fame acquired by fraud obviously "don't count." In _The Fountainhead_, Peter Keating's outwardly successful architecture career has been a sham: he social-engineered his way into partnership in his firm, and all of his best work was plagiarized from the hero, Howard Roark. The turning point for Keating's character is when he asks Roark to let him plagiarize his work one last time, for the Cortlandt housing project, which Roark would never be allowed to work on for political reasons. Keating finally realizes that fraudulent "success" in the eyes of others is no success at all:
+
+> "You'll get everything society can give a man. You'll keep all the money. You'll take any fame or honor anyone might want to grant. You'll accept such gratitude as the tenants might feel. And I—I'll take what nobody can give a man, except himself. I will have built Cortlandt." [said Roark.]
+>
+> "You're getting more than I am, Howard."
+
+In summary, the ultimate sin in Rand's moral universe isn't _giving_ charity. (Because, within the ideology, helping those others whom _you_ want to help, is selfish.) What's evil is _demanding_ charity, claiming the _unearned_, expecting other people to work for your benefit because _you_ supposedly need them to.
+
+## II.
+
+Something people have occasionally noticed about my intellectual style is that I like to win arguments. I take pride and pleasure in pointing out flaws in other people's work in the anticipation of the audience appreciating how clever I am for finding the hole in someone's reasoning.
+
+The people pointing out this fact about me generally seem to think it's a bad thing. They tell me that I should be more charitable to the viewpoints of others, that I ought to be doing [_collaborative_ truth-seeking](https://www.lesswrong.com/posts/uvqd3YiBcrPxXzxQM/what-does-the-word-collaborative-mean-in-the-phrase).
+
+It's true, of course, that there's a terrible danger in wanting to win arguments. [Once your conclusion has been determined, coming up with more arguments for it can't make you more correct](https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-line), even if it can help you "win" a debate. Learning something entails changing your mind, which people are often reluctant to do because it amounts to "losing".
+
+A useful heuristic for overcoming this bias against being willing to "lose" arguments is to take heed of a ["principle of charity"](https://en.wikipedia.org/wiki/Principle_of_charity), of taking the strongest and most rational interpretation of others' words. The person you're arguing against is trying to do what _they_ think is right. If you end up disagreeing with them, it shouldn't be because they're stupid and evil; your theory about why the other person is getting the wrong answer shouldn't make them look _that_ bad. If it does, that's a sign that you haven't really understood their point of view and therefore can't claim to have justly refuted it.
+
+From the standpoint of ideal epistemology, however, the "principle of charity" is not a principle, and the idea of "charity" itself is irrelevant or incoherent. Normatively, theories are preferred to [the quantitative extent that they are simple and predict the observed data](https://www.lesswrong.com/posts/mB95aqTSJLNR9YyjH/message-length). There is no concept of a theory "belonging to" someone, or favoring someone's interests.
+
+For contingent evolutionary-psychological reasons, humans are innately biased to prefer "their own" ideas, and in that context, a "principle of charity" can be useful as a corrective heuristic—but the corrective heuristic only works by colliding the non-normative bias with a fairness instinct, effectively playing the bias against itself: you wouldn't like it if someone dismissed "your" ideas without understanding why they appeal to you, goes the thought, so you should extend the same consideration to others.
+
+Normatively, of course, this is nonsense. You should update on an interlocutor's arguments _for the same reason_ that a scientist working alone would update on the results of an experiment: because (and to the extent that) the result [conveys information about reality](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence). We would not speak of being _charitable_ to an experimental apparatus. The scientist is not doing their lab equipment a _favor_.
+
+Because the principle of charity is merely a corrective heuristic for the bias of arbitrarily favoring "one's own" ideas, it correspondingly only makes sense to apply in one direction—as a corrective for _one's own_ thoughts. I tell myself to make a special effort to look for reasons why I might be wrong and my interlocutor is right because, knowing what I do about human nature, I _selfishly_ expect to thereby achieve more accurate beliefs than I would in the absence of the special effort. It's a workaround, a mitigation for a known bug in human cognition; it makes sense whether or not the other person reciprocates, and whether or not I'm particularly trying to collaborate with them.
+
+On the other hand, when someone _who is currently trying to persuade me of something_ tells me that it doesn't look I'm making enough effort to think of reasons why they're right, that immediately makes me think they're more likely to be wrong. Why? Because I think that if they had an argument, they would be telling me the argument, not chastising my lack of charity. The advice to be on special lookout for reasons your interlocutor is right is good in general, but your interlocutor is the last person to be trusted to give it, because (due to the warp in human psychology) they have an ulterior motive.
+
+Overall, when I look at the world of discourse I see, the moral I draw is not that that collaborative truth-seeking is bad.
+
+It's that collaborative truth-seeking _doesn't exist_. The people claiming to be collaborative truth-seekers are _lying_. Given that everyone wants to be seen as right, the question is: are you going to try to be seen as right by means of providing valid evidence and reasoning, or by—other means?
+
+Or to put it another way: the commenter who admits they care for status is all right. They're usually worth the status they earn. They won't lie for it. They don't have to. But watch out for the commenter who yells too loudly how much they scorn status. [Watch out particularly](https://www.lesswrong.com/posts/jrLkMFd88b4FRMwC6/don-t-double-crux-with-suicide-rock) for the one who yells that others must scorn it. They're after something much worse than status.
+
+Furthermore, I know that "winning" a debate via sophistry and rhetorical tricks obviously "doesn't count." Maybe I could fool an undiscriminating audience, but _I_ would know it wasn't real.
+
+Sometimes I want people to understand some _specific_ truth (out of the vast space of possible truths to pay attention to), for selfish reasons of my own. In these cases, I'm happy to do the work of explaining to put it on the shared map. When someone asks me questions about my work, I don't regard it as an attack, because I expect to be able to answer them—and if I can't, that's _my_ problem.
+
+I will never ask my interlocutors to be more charitable to me. I _will_ often say "That's not what I meant", or "That's not a reasonable interpretation of the text I published"—but that's a claim about what _I_ mean, or a claim about the text; it's not a claim _on them_. I don't expect people to listen to me because _I_ supposedly need them to.
+
+## III.
+
+My favorite scene in _Atlas Shrugged_ is the one where Cherryl Taggart (née Brooks) goes to see Dagny Taggart after discovering the truth about her marriage. Cherryl had married Dagny's brother James thinking that he was the intrepid industrialist responsible for the success of the Taggart Transcontinental railroad, only to later find out that James is a phony political actor who took credit for Dagny's accomplishments after the fact, despite having opposed her initiatives and made her work more difficult.
+
+("I married Jim because I ... I thought that he was _you_," Cherryl tells Dagny. There is some very beautiful slash fanfiction that needs to be written picking up from that line, which is out of scope for this blog post.)
+
+Cherryl intends only to briefly apologize to Dagny for earlier insulting remarks, not to make any further imposition—and is surprised when Dagny not only forgives her, but seems to take a genuine interest in her welfare. It's worth quoting at length:
+
+> "You've had a terrible time, haven't you?" [said Dagny.]
+>
+> "Yes ... but that doesn't matter ... that's my own problem ... and my own fault."
+>
+> "I don't think it was your own fault."
+>
+> Cherryl did not answer, then said suddenly, desperately, "Look ... what I don't want is charity."
+>
+> "Jim must have told you—and it's true—that I never engage in charity."
+>
+> "Yes, he did ... But what I mean is—"
+>
+> "I know what you mean."
+>
+> "But there's no reason why you should have to feel concern for me ... I didn't come here to complain and ... and load another burden on your shoulders. ... That I happen to suffer, doesn't give me a claim on you."
+>
+> "No, it doesn't. But that you value all the things I value, does."
+>
+> "You mean ... if you want to talk to me, it's not alms? Not just because you feel sorry for me?"
+>
+> "I feel terribly sorry for you, Cherryl, and I'd like to help you—not because you suffer, but because you haven't deserved to suffer."
+>
+> "You mean, you wouldn't be kind to anything weak or whining or rotten about me? Only to whatever you see in me that's good?"
+>
+> "Of course."
+>
+> Cherryl did not move her head, but she looked as if it were lifted—as if some bracing current were relaxing her features into that rare look which combines pain and dignity.
+>
+> "It's not alms, Cherryl. Don't be afraid to speak to me."
+>
+> [...]
+>
+> "You know, Miss Tag—Dagny," she said softly, in wonder, "you're not as I expected you to be at all. ... They, Jim and his friends, they said you were hard and cold and unfeeling."
+>
+> "But it's true, Cherryl, I _am_, in the sense they mean—only have they ever told you in just what sense they mean it?"
+>
+> "No. They never do. They only sneer at me when I ask them what they mean by anything ... about anything. What did they mean about you?"
+>
+> "Whenever anyone accuses some person of being 'unfeeling', he means that that person is just. He means that that person has no causeless emotions and will not grant him a feeling which he does not deserve. He means that 'to feel' is to go against reason, against moral values, against reality. He means ... What's the matter?" she asked, seeing the abnormal intensity of the girl's face.
+>
+> "It's ... it's something I've tried so hard to understand ... for such a long time. ..."
+>
+> "Well, observe that you never hear that accusation in defense of innocence, but always in defense of guilt. You never hear it said by a good person about those who fail to do him justice. But you always hear it said by a rotter about those who treat him as a rotter, those who don't feel any sympathy for the evil he's committed or for the pain he suffers as a consequence. Well, it's true—_that_ is what I do not feel. But those who feel it, feel nothing for any quality of human greatness, for any person or action that deserves admiration, approval, esteem. _These_ are the things _I_ feel. You'll find that it's one or the other. Those who grant sympathy to guilt, grant none to innocence. Ask yourself which, of the two, are the _unfeeling_ persons. And then you'll see what motive is the opposite of charity."
+>
+> "What?" she whispered.
diff --git a/content/2023/lack-of-social-grace-is-an-epistemic-virtue.md b/content/2023/lack-of-social-grace-is-an-epistemic-virtue.md
new file mode 100644
index 0000000..57e72ce
--- /dev/null
+++ b/content/2023/lack-of-social-grace-is-an-epistemic-virtue.md
@@ -0,0 +1,88 @@
+Title: Lack of Social Grace Is an Epistemic Virtue
+Date: 2023-07-31 09:38
+Status: published
+Category: philosophy
+Tags: rationality
+Slug: lack-of-social-grace-is-an-epistemic-virtue
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/h2Hk2c2Gp5sY4abQh/lack-of-social-grace-is-an-epistemic-virtue)
+
+Someone once told me that they thought I acted like refusing to employ the bare minimum of social grace was a virtue, and that this was bad. (I'm paraphrasing; they actually used a different word that starts with _b_.)
+
+I definitely don't want to say that lack of social grace is _unambiguously_ a virtue. Humans are social animals, so the set of human virtues is almost certainly going to involve doing social things gracefully!
+
+Nevertheless, I will bite the bullet on a weaker claim. Politeness is, to a large extent, about concealing or obfuscating information that someone would prefer not to be revealed—that's why we recognize the difference between one's honest opinion, and what one says when one is "just being polite." Idealized honest Bayesian reasoners would not have social graces—and therefore, humans trying to imitate idealized honest Bayesian reasoners will tend to bump up against (or smash right through) the bare minimum of social grace. In this sense, we might say that the lack of social grace is an "epistemic" virtue—even if it's probably not great for normal humans trying to live normal human lives.
+
+Let me illustrate what I mean with one fictional and one real-life example.
+
+------
+
+The beginning of the film _The Invention of Lying_ (before the eponymous invention of lying) depicts an alternate world in which everyone is [radically honest](https://www.lesswrong.com/posts/GMhzDb3uAFYLwmXtY/radical-honesty)—not just in [the narrow sense of not lying](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly), but more broadly saying exactly what's on their mind, without thought of concealment.
+
+In [one scene](https://www.youtube.com/watch?v=3DmchoOLczY), our everyman protagonist is on a date at a restaurant with an attractive woman.
+
+"I'm very embarrassed I work here," says the waiter. "And you're very pretty," he tells the woman. "That only makes this worse."
+
+"Your sister?" the waiter then asks our protagonist.
+
+"No," says our everyman.
+
+"Daughter?"
+
+"No."
+
+"She's way out of your league."
+
+"... thank you."
+
+The woman's cell phone rings. She explains that it's her mother, probably calling to check on the date.
+
+"Hello?" she answers the phone—still at the table, with our protagonist hearing every word. "Yes, I'm with him right now. ... No, not very attractive. ... No, doesn't make much money. It's alright, though, seems nice, kind of funny. ... A bit fat. ... Has a funny little—snub nose, kind of like a frog in the—facial ... No, I won't be sleeping with him tonight. ... No, probably not even a kiss. ... Okay, you too, 'bye."
+
+The scene is [funny because](https://en.wikipedia.org/wiki/Cringe_comedy) of how it violates the expected social conventions of our own world. In our world, politeness demands that you not say negative-valence things about someone in front of them, because people don't like hearing negative-valence things about themselves. Someone in our world who behaved like the woman in this scene—calling someone ugly and poor and fat right in front of them—could only be acting out of deliberate cruelty.
+
+But the people in the movie _aren't like us_. Having taken the call, why _should_ she speak any differently just because the man she was talking about could hear? Why would he object? To a decision-theoretic agent, the [value of information](https://en.wikipedia.org/wiki/Value_of_information) is always nonnegative. Given that his date thought he was unattractive, how could it be worse for him to know rather than not-know?
+
+For humans from our world, these questions do have answers—_complicated_ answers having to do with things like map–territory confusions that make receiving bad news seem like a bad event (rather than the good event of learning information about how things were _already_ bad, whether or not you knew it), and how it's advantageous for others to have positive-valence false beliefs about oneself.
+
+The world of _The Invention of Lying_ is simpler, clearer, easier to navigate than our world. There, you don't have to _worry_ whether people don't like you and are planning to harm your interests. They'll tell you.
+
+------
+
+In ["Los Alamos From Below"](https://archive.is/3DKrJ), physicist Richard Feynman's account of his work on the [Manhattan Project](https://en.wikipedia.org/wiki/Manhattan_Project) to build the first atomic bomb, Feynman recalls being sought out by a much more senior physicist specifically for his lack of social graces:
+
+> I also met Niels Bohr. His name was Nicholas Baker in those days, and he came to Los Alamos with Jim Baker, his son, whose name is really Aage Bohr. They came from Denmark, and they were _very_ famous physicists, as you know. Even to the big shot guys, Bohr was a great god.
+>
+> We were at a meeting once, the first time he came, and everybody wanted to _see_ the great Bohr. So there were a lot of people there, and we were discussing the problems of the bomb. I was back in a corner somewhere. He came and went, and all I could see of him was from between people's heads.
+>
+> In the morning of the day he's due to come next time, I get a telephone call.
+>
+> "Hello—Feynman?"
+>
+> "Yes."
+>
+> "This is Jim Baker." It's his son. "My father and I would like to speak to you."
+>
+> "Me? I'm Feynman, I'm just a—"
+>
+> "That's right. Is eight o'clock OK?"
+>
+> So, at eight o'clock in the morning, before anybody's awake, I go down to the place. We go into an office in the technical area and he says, "We have been thinking how we could make the bomb more efficient and we think of the following idea."
+>
+> I say, "No, it's not going to work. It's not efficient ... Blah, blah, blah."
+>
+> So he says, "How about so and so?"
+>
+> I said, "That sounds a little bit better, but it's got this damn fool idea in it."
+>
+> This went on for about two hours, going back and forth over lots of ideas, back and forth, arguing. [...]
+>
+> "Well," [Niels Bohr] said finally, lighting his pipe, "I guess we can call in the big shots _now_." So then they called all the other guys and had a discussion with them.
+>
+> Then the son told me what happened. The last time he was there, Bohr said to his son, "Remember the name of that little fellow in the back over there? He's the only guy who's not afraid of me, and will say when I've got a crazy idea. So the _next_ time when we want to discuss ideas, we're not going to be able to do it with these guys who say everything is yes, yes, Dr. Bohr. Get that guy and we'll talk with him first."
+>
+> I was always _dumb_ in that way. I never knew who I was talking to. I was always worried about the physics. If the idea looked lousy, I said it looked lousy. If it looked good, I said it looked good. Simple proposition.
+
+Someone who felt uncomfortable with Feynman's bluntness and wanted to believe that there's no conflict between rationality and social graces might argue that Feynman's "simple proposition" is actually wrong insofar as it fails to appreciate the map–territory distinction: in saying, "No, it's not going to work", was not Feynman implicitly asserting that just because _he_ couldn't see a way to make it work, it simply couldn't? And in general, _shouldn't_ you know who you're talking to? Wasn't Bohr, the Nobel prize winner, more likely to be right than Feynman, the fresh young Ph.D. [(at the time)](https://en.wikipedia.org/wiki/Richard_Feynman#Manhattan_Project)?
+
+While not entirely without merit (it's true that the map is not the territory; it's true that authority is not without evidential weight), attending overmuch to such nuances distracts from [_worrying about the physics_](https://www.lesswrong.com/posts/5yFRd3cjLpm3Nd6Di/argument-screens-off-authority), which is what Bohr _wanted_ out of Feynman—and, incidentally, what _I_ want out of my readers. I would not expect readers to confirm interpretations with me before publishing a critique. If the post looks lousy, say it looks lousy. If it looks good, say it looks good. Simple proposition.
diff --git a/content/2023/rationalist-discourse-is-like-physicist-motors.md b/content/2023/rationalist-discourse-is-like-physicist-motors.md
new file mode 100644
index 0000000..54d170e
--- /dev/null
+++ b/content/2023/rationalist-discourse-is-like-physicist-motors.md
@@ -0,0 +1,86 @@
+Title: “Rationalist Discourse” Is Like “Physicist Motors”
+Date: 2023-02-25 21:58
+Status: published
+Category: philosophy
+Tags: rationality, discourse
+Slug: rationalist-discourse-is-like-physicist-motors
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/SX6wQEdGfzz7GKYvp/rationalist-discourse-is-like-physicist-motors)
+
+Imagine being a student of physics, and coming across a blog post proposing a list of guidelines for "physicist motors"—motor designs informed by the knowledge of physicists, unlike ordinary motors.
+
+Even if most of the things on the list seemed like sensible advice to keep in mind when designing a motor, the framing would seem very odd. The laws of physics describe how energy can be converted into work. To the extent that _any_ motor accomplishes anything, it happens within the laws of physics. There are theoretical ideals describing how motors need to work in principle, like [the Carnot engine](https://en.wikipedia.org/wiki/Carnot_heat_engine), but you can't actually build an ideal Carnot engine; real-world electric motors or diesel motors or jet engines all have their own idiosyncratic lore depending on the application and the materials at hand; an engineer who worked on one, might not the be best person to work on another. You might appeal to principles of physics to explain why some particular motor is inefficient or poorly-designed, but you would not speak of _physicist motors_ as if that were a distinct category of thing—and if someone _did_, you might quietly begin to doubt how much they really knew about physics.
+
+As a [student of rationality](https://twitter.com/ESYudkowsky/status/1355711473490685952), I feel the same way about guidelines for "rationalist discourse." The laws of probability and decision theory describe how information can be converted into optimization power. To the extent that _any_ discourse accomplishes anything, [it happens within the laws of rationality](https://www.lesswrong.com/posts/eY45uCCX7DdwJ4Jha/no-one-can-exempt-you-from-rationality-s-laws).
+
+Rob Bensinger proposes ["Elements of Rationalist Discourse"](https://www.lesswrong.com/posts/svuBpoSduzhYjFPrA/elements-of-rationalist-discourse) as a companion to Duncan Sabien's earlier ["Basics of Rationalist Discourse"](https://www.lesswrong.com/posts/XPv4sYrKnPzeJASuk/basics-of-rationalist-discourse-1). _Most_ of the things on both lists are, indeed, sensible advice that one might do well to keep in mind when arguing with people, but as Bensinger notes, "Probably this new version also won't match '_the_ basics' as other people perceive them."
+
+But there's a reason for that: a list of guidelines has the wrong type signature for being "_the_ basics". The _actual_ basics are the principles of rationality one would appeal to _explain which guidelines are a good idea_: principles like how [evidence is the systematic correlation between possible states of your observations and possible states of reality](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence), how [you need evidence to locate the correct hypothesis in the space of possibilities](https://www.lesswrong.com/posts/nj8JKFoLSMEmD3RGp/how-much-evidence-does-it-take), how [the quality of your conclusion can only be improved by arguments that have the power to _change_ that conclusion](https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-line).
+
+Contemplating these basics, it should be clear that there's just not going to be anything like a unique style of "rationalist discourse", any more than there is a unique "physicist motor." There are theoretical ideals describing how discourse needs to work in principle, like [Bayesian reasoners with common priors exchanging probability estimates](https://en.wikipedia.org/wiki/Aumann's_agreement_theorem), but you can't actually build an ideal Bayesian reasoner. Rather, different discourse algorithms (the collective analogue of ["cognitive algorithm"](https://www.lesswrong.com/posts/HcCpvYLoSFP4iAqSz/rationality-appreciating-cognitive-algorithms)) leverage the laws of rationality to convert information into optimization in somewhat different ways, depending on the application and the population of interlocutors at hand, much as electric motors and jet engines both leverage the laws of physics to convert energy into work without being identical to each other, and with each requiring their own engineering sub-specialty to design.
+
+Or to use [another classic metaphor](https://www.lesswrong.com/posts/teaxCFgtmCQ3E9fy8/the-martial-art-of-rationality), there's also just not going to be a unique martial art. Boxing and karate and ju-jitsu all have their own idiosyncratic lore adapted to different combat circumstances, and a master of one would easily defeat a novice of the other. One might appeal to the laws of physics and the properties of the human body to explain why some particular martial arts school was not teaching their students to fight effectively. But if some particular karate master were to brand their own lessons as the "basics" or "elements" of "martialist fighting", you might quietly begin to doubt how much actual fighting they had done: either all fighting is "martialist" fighting, or "martialist" fighting isn't actually necessary for beating someone up.
+
+One historically important form of discourse algorithm is _debate_, and its close variant the _adversarial court system_. It works by separating interlocutors into two groups: one that searches for arguments in favor of a belief, and another that searches for arguments against the belief. Then anyone listening to the debate can consider all the arguments to help them decide whether or not to adopt the belief. (In the _court_ variant of debate, a designated "judge" or "jury" announces a "verdict" for or against the belief, which is added to the court's [shared map](https://www.lesswrong.com/posts/9QxnfMYccz9QRgZ5z/the-costly-coordination-mechanism-of-common-knowledge), where it can be referred to in subsequent debates, or "cases.")
+
+The enduring success and legacy of the debate algorithm can be attributed to how it circumvents a critical design flaw in individual human reasoning, the tendency to "rationalize"—to preferentially search for new arguments for an already-determined conclusion.
+
+(At least, "design flaw" is one way of looking at it—a more complete discussion would consider how individual human reasoning capabilities _co-evolved_ with the debate algorithm—and, as I'll briefly discuss later, this "bug" for the purposes of reasoning is actually a "feature" for the purposes of deception.)
+
+As a consequence of rationalization, once a conclusion has been reached, even prematurely, further invocations of the biased argument-search process are likely to further entrench the conclusion, even when strong counterarguments exist (in regions of argument-space neglected by the biased search). The debate algorithm solves this sticky-conclusion bug by distributing a search for arguments and counterarguments among multiple humans, [ironing out falsehoods](https://www.lesswrong.com/posts/iThwqe3yPog56ytyq/aiming-for-convergence-is-like-discouraging-betting) by pitting two biased search processes against each other. (For readers more familiar with artificial than human intelligence, [generative adversarial networks](https://en.wikipedia.org/wiki/Generative_adversarial_network) work on a similar principle.)
+
+For all its successes, the debate algorithm also suffers from many glaring flaws. For one example, the benefits of improved conclusions mostly accrue to third parties who haven't already entrenched on a conclusion; debate participants themselves are [rarely seen changing their minds](https://www.lesswrong.com/posts/buixYfcXBah9hbSNZ/we-change-our-minds-less-often-than-we-think). For another, just the choice of what position to debate has a distortionary effect even on the audience; if [it takes more bits to _locate_ a hypothesis for consideration than to convincingly confirm or refute it](https://www.lesswrong.com/posts/MwQRucYo6BZZwjKE7/einstein-s-arrogance), then most of the relevant cognition has already happened by the time people are arguing for or against it. Debate is also inefficient: for example, if the "defense" in the court variant happens to find evidence or arguments that would benefit the "prosecution", the defense has no incentive to report it to the court, and there's no guarantee that the prosecution will independently find it themselves.
+
+Really, the whole idea is so galaxy-brained that it's amazing it works at all. There's only one reality, so correct information-processing should result in everyone agreeing on the best, most-informed belief-state. [This is formalized in Aumann's famous agreement theorem](https://en.wikipedia.org/wiki/Aumann's_agreement_theorem), but even without studying the proofs, the result is _obvious_. A generalization to a more realistic setting without instantaneous communication gives the result that [disagreements should be unpredictable](http://mason.gmu.edu/~rhanson/unpredict.pdf): after Bob the Bayesian tells Carol the Coherent Reasoner his belief, Bob's expectation of the difference between his belief and Carol's new belief should be zero. (That is, Carol might still disagree, but Bob shouldn't be able to predict whether it's in the same direction as before, or whether Carol now holds a _more_ extreme position on what adherents to the debate algorithm would call "Bob's side.")
+
+That being the normative math, why does the human world's enduringly dominant discourse algorithm take for granted the ubiquity of, not just disagreements, but _predictable_ disagreements? Isn't that crazy?
+
+Yes. It is crazy. One might hope to do better by developing some sort of training or discipline that would allow discussions between practitioners of such "rational arts" to depart from the harnessed insanity of the debate algorithm with its stubbornly stable "sides", and instead mirror the side-less Bayesian ideal, the free flow of all available evidence channeling interlocutors to an unknown destination.
+
+Back in late 'aughts, an attempt to articulate what such a discipline might look like was published on a blog called _Overcoming Bias_. (You probably haven't heard of it.) It's been well over a decade since then. How is that going?
+
+Eliezer Yudkowsky [laments](https://www.lesswrong.com/posts/7im8at9PmhbT4JHsW/ngo-and-yudkowsky-on-alignment-difficulty):
+
+> In the end, a lot of what people got out of all that writing I did, was not the deep object-level principles I was trying to point to—they did not really get [Bayesianism as thermodynamics](https://www.lesswrong.com/posts/QkX2bAkwG2EpGvNug/the-second-law-of-thermodynamics-and-engines-of-cognition), say, they did not become able to see [Bayesian structures](https://www.lesswrong.com/posts/QrhAeKBkm2WsdRYao/searching-for-bayes-structure) any time somebody sees a thing and changes their belief. What they got instead was something much more meta and general, a vague spirit of how to reason and argue, because that was what they'd spent a lot of time being exposed to over and over and over again in lots of blog posts.
+
+"A vague spirit of how to reason and argue" seems like an apt description of what "Basics of Rationalist Discourse" and "Elements of Rationalist Discourse" are attempting to codify—but with no explicit instruction on which guidelines arise from deep object-level principles of normative reasoning, and which from mere taste, politeness, or adaptation to local circumstances, it's unclear whether students of 2020s-era "rationalism" are poised to significantly outperform the traditional debate algorithm—and it seems alarmingly possible to do _worse_, if [the collaborative aspects of modern "rationalist" discourse allow participants to introduce errors](https://www.lesswrong.com/posts/jrLkMFd88b4FRMwC6/don-t-double-crux-with-suicide-rock) that a designated adversary under the debate algorithm would have been incentivized to correct, and most "rationalist" practitioners don't have a deep theoretical understanding of _why debate works_ as well as it does.
+
+Looking at Bensinger's "Elements", there's a clear-enough connection between the first eight points (plus three sub-points) and the laws of normative reasoning. Truth-Seeking, Non-Deception, and Reality-Minding, trivial. Non-Violence, because violence doesn't distinguish between truth and falsehood. Localizability, in that I can affirm the [validity](https://www.lesswrong.com/posts/WQFioaudEH8R7fyhm/local-validity-as-a-key-to-sanity-and-civilization) of an argument that A would imply B, while simultaneously denying A. Alternative-Minding, because decisionmaking under uncertainty requires living in many possible worlds. And so on. (Lawful justifications for the elements of Reducibility and Purpose-Minding left as an exercise to the reader.)
+
+But then we get this:
+
+> 9. **Goodwill.** Reward others' good epistemic conduct (_e.g._, updating) more than most people naturally do. Err on the side of carrots over sticks, forgiveness over punishment, and civility over incivility, unless someone has explicitly set aside a weirder or more rough-and-tumble space.
+
+I can believe that these are good ideas for having a pleasant conversation. But separately from whether "Err on the side of forgiveness over punishment" is a good idea, it's hard to see how it belongs on the _same_ list as things like "Try not to 'win' arguments using [...] tools that work similarly well whether you're right or wrong" and "[A]sk yourself what Bayesian evidence you have that you're not in those alternative worlds".
+
+The difference is this. If your discourse algorithm lets people "win" arguments with tools that work equally well whether they're right or wrong, then your discourse _gets the wrong answer_ (unless, by coincidence, the people who are best at winning are also the best at getting the right answer). If the interlocutors in your discourse don't ask themselves what Bayesian evidence they have that they're not in alternative worlds, then your discourse _gets the wrong answer_ (if you happen to live in an alternative world).
+
+If your discourse algorithm errs on the side of sticks over carrots (perhaps, emphasizing _punishing_ others' _bad_ epistemic conduct more than most people naturally do), then ... what? How, specifically, are rough-and-tumble spaces less ["rational"](https://www.lesswrong.com/posts/hN8Ld8YdqFsui2xgc/only-say-rational-when-you-can-t-eliminate-the-word), more prone to _getting the wrong answer_, such that a list of "Elements of Rationalist Discourse" has the authority to designate them as non-default?
+
+I'm not saying that goodwill is _bad_, particularly. I totally believe that goodwill is a necessary part of many discourse algorithms that produce maps that reflect the territory, much like how kicking is a necessary part of many martial arts (but not boxing). It just seems like a bizarre thing to put in a list of guidelines for "rationalist discourse".
+
+It's as if guidelines for designing "physicist motors" had a point saying, "Use more pistons than most engineers naturally do." It's not that pistons are bad, particularly. Lots of engine designs use pistons! It's just, the pistons are there specifically to convert force from expanding gas into rotational motion. I'm pretty pessimistic about the value of attempts to teach junior engineers to mimic the surface features of successful engines without teaching them how engines work, even if the former seems easier.
+
+The example given for "[r]eward[ing] others' good epistemic conduct" is "updating". If your list of "Elements of Rationalist Discourse" is _just_ trying to apply a [toolbox](https://www.lesswrong.com/posts/CPP2uLcaywEokFKQG/toolbox-thinking-and-law-thinking) of directional nudges to improve the median political discussion on social media (where everyone is yelling and no one is thinking), then sure, directionally nudging people to directionally nudge people to look like they're updating probably is a directional improvement. It still seems awfully unambitious, compared to trying to teach the _criteria by which_ we can tell it's an improvement. In some contexts (in-person interactions with someone I like or respect), I think [I have the opposite problem](https://slatestarcodex.com/2014/03/24/should-you-reverse-any-advice-you-hear/), of being disposed to agree with the person I'm currently talking to, in a way that shortcuts the slow work of grappling with their arguments and doesn't stick after I'm not talking to them anymore; I look as if I'm "updating", but I haven't actually _learned_. Someone who thought "rationalist discourse" entailed "[r]eward[ing] others' good epistemic conduct (_e.g._, updating) more than most people naturally do" and sought to act on me accordingly would be making that problem _worse_.
+
+A footnote on the "Goodwill" element elaborates:
+
+> Note that this doesn't require assuming everyone you talk to is honest or has good intentions.
+>
+> It does have some overlap with the rule of thumb "as a very strong but defeasible default, carry on object-level discourse as if you were role-playing being on the same side as the people who disagree with you".
+
+But this seems to contradict the element of Non-Deception. If you're _not_ actually on the same side as the people who disagree with you, why would you (as a very strong but defeasible default) role-play otherwise?
+
+Other intellectual communities have a name for the behavior of role-playing being on the same side as people you disagree with: they call it ["concern trolling"](https://geekfeminism.fandom.com/wiki/Concern_troll), and they think it's a _bad_ thing. Why is that? Are they just less rational than "us", the "rationalists"?
+
+Here's what I think is going on. There's another aspect to the historical dominance of the debate algorithm. The tendency to rationalize new arguments for a fixed conclusion is only a bug if one's goal is to improve the conclusion. If the fixed conclusion was adopted for _other_ reasons—notably, because one would benefit from other people believing it—then generating new arguments might help persuade those others. If persuading others is the real goal, then rationalization is _not_ irrational; it's just dishonest. (And if one's concept of "honesty" is [limited to not consciously making false statements](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly), it might not even be dishonest.) Society benefits from using the debate algorithm to improve shared maps, but most individual debaters are mostly focused on getting their preferred beliefs onto the shared map.
+
+That's why people don't like concern trolls. If my faction is trying to get Society to adopt beliefs that benefit our faction onto the shared map, someone who comes to us role-playing being on our side, but who is actually trying to stop us from adding our beliefs to the shared map just because they think our beliefs don't reflect the territory, isn't a friend; they're a double agent, an enemy _pretending_ to be a friend, which is worse than the honest enemy we expect to face before the judge in the debate hall.
+
+This vision of factions warring to make Society's shared map benefit themselves is pretty bleak. It's tempting to think the whole mess could be fixed by starting a new faction—the "rationalists"—that is solely dedicated to making Society's shared map reflect the territory: a culture of clear thinking, clear communication, and collaborative truth-seeking.
+
+I don't think it's that simple. You _do_ have interests, and if you can fool yourself into thinking that you don't, your competitors are unlikely to fall for it. Even if your claim to only want Society's shared map to reflect the territory were true—which it isn't—_anyone could just say that_.
+
+[I don't immediately have solutions on hand.](https://www.lesswrong.com/posts/uHYYA32CKgKT3FagE/hold-off-on-proposing-solutions) Just an intuition that, if there _is_ any way of fixing this mess, it's going to involve clarifying conflicts rather than obfuscating them—looking for Pareto improvements, rather than pretending that everyone has the same utility function. That if something called "rationalism" is to have any value whatsoever, it's as the _field of study_ that can do things like _explain why it makes sense that people don't like concern trolling_. Not as as its own faction with its own weird internal social norms that call for concern trolling as a very strong but defeasible default.
+
+But don't take my word for it.
diff --git a/content/2023/youll-never-persuade-people-like-that.md b/content/2023/youll-never-persuade-people-like-that.md
new file mode 100644
index 0000000..07d74c0
--- /dev/null
+++ b/content/2023/youll-never-persuade-people-like-that.md
@@ -0,0 +1,36 @@
+Title: “You’ll Never Persuade People Like That”
+Date: 2023-03-11 21:38
+Status: published
+Category: philosophy
+Tags: rationality, discourse
+Slug: youll-never-persuade-people-like-that
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/67NrgoFKCWmnG3afd/you-ll-never-persuade-people-like-that)
+
+Sometimes, when someone is arguing for some proposition, their interlocutor will reply that the speaker's choice of arguments or tone wouldn't be effective at persuading some third party.
+
+This would seem to be an odd change of topic. If I was arguing for this-and-such proposition, and my interlocutor isn't, themselves, convinced by my arguments, it makes sense for them to reply about why they, personally, aren't convinced. Why is it relevant whether I would convince some third party that _isn't here_?
+
+What's going on in this kind of situation? Why would someone think "You'll never persuade people like that" was a relevant reply?
+
+"Because people aren't truthseeking and treat arguments as soldiers" doesn't seem like an adequate explanation by itself. It's true, but it's not specific enough: what particularly makes appeal-to-persuading-third-parties an effective "soldier"?
+
+------
+
+The [bargaining model of war](https://en.wikipedia.org/wiki/Bargaining_model_of_war) attempts to explain why wars are fought—and _not_ fought; even the bitterest enemies often prefer to grudgingly make peace with each other rather than continue to fight.
+
+That's because war is costly. If I estimate that by continuing to wage war, there's a 60% chance my armies will hold a desirable piece of territory, I can achieve my war objectives equally well [in expectation](https://en.wikipedia.org/wiki/Expected_value)—while saving a lot of money and human lives—by instead signing a peace treaty that divides the territory with the enemy 60/40.
+
+If the enemy will agree to that, of course. The enemy has their own forecast probabilities and their own war objectives. There's usually a range of possible treaties that both combatants will prefer to fighting, but the parties need to negotiate to select a particular treaty, because there's typically no uniquely obvious "fair" treaty—similar to how a buyer and seller need to negotiate a price for a rare and expensive item for which there is no uniquely obvious "fair" price.
+
+-----
+
+If war is bargaining, and arguments are soldiers, then debate is negotiation: the same game-theoretic structure shines through armies fighting over the borders on the world's political map, buyer and seller haggling over contract items, and debaters arguing over the beliefs on Society's shared map. Strong arguments, like a strong battalion, make it less tenable for the adversary to maintain their current position.
+
+Unfortunately, the theory of interdependent decision is ... subtle. Although [recent work points toward the outlines of a more elegant theory](https://arbital.com/p/logical_dt/) with fewer pathologies, the classical understanding of negotiation often recommends "rationally irrational" tactics in which an agent handicaps its own capabilities in order to extract concessions from a counterparty: for example, in the deadly game of [chicken](https://en.wikipedia.org/wiki/Chicken_(game)), if I visibly throw away my steering wheel, oncoming cars are forced to swerve for me in order to avoid a crash, but if the oncoming drivers have already blindfolded themselves, they wouldn't be able to _see_ me throw away my steering wheel, and I am forced to swerve for them.
+
+Thomas Schelling teaches us that one such tactic is to _move the locus of the negotiation elsewhere_, onto some third party who has less of an incentive to concede or is less able to be communicated with. For example, if business purchases over $500 have to be approved by my hard-to-reach boss, an impatient seller of an item that ordinarily goes for $600 might be persuaded to give me a discount.
+
+And that's what explains the attractiveness of the appeal-to-persuading-third-parties. What "You'll never persuade people like that" _really_ means is, "You are starting to persuade _me_ against my will, and I'm laundering my cognitive dissonance by asserting that you actually need to persuade someone else who isn't here." When someone is desperate enough to try to get away with that, you _know_ you've got them cornered. Go for the throat!
+
+(Unless the belief you're arguing for is false. You checked that beforehand, right??)
diff --git a/content/2024/and-all-the-shoggoths-merely-players.md b/content/2024/and-all-the-shoggoths-merely-players.md
new file mode 100644
index 0000000..fda35f6
--- /dev/null
+++ b/content/2024/and-all-the-shoggoths-merely-players.md
@@ -0,0 +1,146 @@
+Title: And All the Shoggoths Merely Players
+Date: 2024-02-10 11:56
+Status: published
+Category: philosophy
+Tags: artificial intelligence, rationality
+Slug: and-all-the-shoggoths-merely-players
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/8yCXeafJo67tYe5L4/and-all-the-shoggoths-merely-players)
+
+_[Setting: a suburban house. The interior of the house takes up most of the stage; on the audience's right, we see a wall in cross-section, and a front porch. **Simplicia** enters stage left and rings the doorbell.]_
+
+**Doomimir**: _[opening the door]_ Well? What do you want?
+
+**Simplicia**: I can't stop thinking about [our last conversation](https://www.lesswrong.com/posts/pYWA7hYJmXnuyby33/alignment-implications-of-llm-successes-a-debate-in-one-act). It was kind of all over the place. If you're willing, I'd like to continue, but focusing in narrower detail on a couple points I'm still confused about.
+
+**Doomimir**: And why should I bother tutoring an Earthling in alignment theory? If you didn't get it from the empty string, and you didn't get it from our last discussion, why should I have any hope of you learning this time? And even if you did, [what good would it do?](https://www.lesswrong.com/posts/4Gcz3fGcYmmzhozxr/hashing-out-long-standing-disagreements-seems-low-value-to)
+
+**Simplicia**: _[serenely]_ If the world is ending either way, I think it's more dignified that I understand exactly why. _[A beat.]_ Sorry, that doesn't explain what's in it for you. That's why I had to ask.
+
+**Doomimir**: _[grimly]_ As you say. If this world is ending either way.
+
+_[He motions for her to come in, and they sit down.]_
+
+**Doomimir**: What are you confused about? I mean, that you wanted to talk about.
+
+**Simplicia**: You seemed really intent on a particular intuition pump against human-imitation-based alignment strategies, where you compared LLMs to an alien actress. I didn't find that compelling.
+
+**Doomimir**: But you claim to understand that LLMs that emit plausibly human-written text aren't human. Thus, the AI is not the character it's playing. Similarly, being able to predict the conversation in a bar, doesn't make you drunk. What's there not to get, even for you?
+
+**Simplicia**: Why doesn't the "predicting barroom conversation doesn't make you drunk" analogy falsely imply "predicting the answers to modular arithmetic problems doesn't mean you implement modular arithmetic"?
+
+**Doomimir**: To predict the conversation in a bar, you need to know everything the drunk people know, separately and in addition to everything you know. Being drunk yourself [would just get in the way](https://twitter.com/ESYudkowsky/status/1744061053754032634). Similarly, predicting the behavior of nice people isn't the same thing as being nice. Modular arithmetic isn't like that; there's nothing besides the knowledge to not implement.
+
+**Simplicia**: But we only need our AI to compute nice behavior, not necessarily to have some internal structure corresponding to the _quale_ of niceness. As far as safety properties go, we don't care whether the actress is "really drunk" as long as she stays in character.
+
+**Doomimir**: _[scoffing]_ Have you tried imagining any internal mechanisms at all other than a bare, featureless inclination to emit the outward behavior you observe?
+
+**Simplicia**: _[unfazed]_ Sure, let's talk about internal mechanisms. The reason I chose modular arithmetic as an example is because it's a task for which we have [good interpretability results](https://arxiv.org/abs/2301.05217). Train a shallow transformer on a subset of the addition problems modulo some fixed prime. The network learns to map the inputs onto a circle in the embedding space, and then does some trigonometry to extract the [residue](https://en.wikipedia.org/wiki/Modular_arithmetic#Congruence_classes), much as one would count forward on the face of an analog clock.
+
+Alternatively, with a slightly different architecture that has a harder time with trig, [it can learn a different algorithm](https://arxiv.org/abs/2306.17844): the embeddings are still on a circle, but the answer is computed by looking at the average of the embedding vectors of the inputs. On the face of an analog clock, the internal midpoints between distinct numbers that sum to 6 mod 12—that's 2 and 4, or 1 and 5, or 6 and 12, or 10 and 8, or 11 and 7—all lie on the line connecting 3 and 9. Thus, the sum-mod-_p_ of two numbers can be determined by which line the midpoint of the inputs falls on—as long as the inputs aren't on opposite sides of the circle, in which case their midpoint is in the center, where all the lines meet. But the network compensates for such antipodal points by also learning another circle in a different subspace of the embedding space, such that inputs that are antipodal on the first circle are close together on the second, which helps disambiguate the answer.
+
+**Doomimir**: Cute results. Excellent work—by Earth standards. And entirely unsurprising. Sure, if you train your neural net on a well-posed mathematical problem with a consistent solution, it will converge on a solution to that problem. What's your point?
+
+**Simplicia**: It's evidence about the feasibility of learning desired behavior from training data. You seem to think that it's hopelessly naïve to imagine that training on "nice" data could result in generalizably nice behavior—that the only reason someone might think that was a viable path was is if they were engaging in [magical reasoning about surface similarities](https://www.lesswrong.com/posts/6ByPxcGDhmx74gPSm/surface-analogies-and-deep-causes). I think it's germane to point out that at least for this toy problem, we have a pretty concrete, non-magical story about how optimizing against a training set discovers an algorithm that reproduces the training data and also generalizes correctly to the test set.
+
+For non-toy problems, we know empirically that deep learning _can_ hit very precise behavioral targets: the vast hypermajority of programs don't speak fluent English or generate beautiful photorealistic images, and yet GPT-4 and Midjourney exist.
+
+If doing _that_ for "text" and "images" was a mere engineering problem, I don't see what fundamental theoretical barrier rules out the possibility of pulling off the same kind of thing for "friendly and moral real-world decisionmaking"—learning a "good person" or "obedient servant" function from data, much as Midjourney has learned a "good art" function.
+
+It's true that diffusion models don't work like a human artist on the inside, but it's not clear why that matters? It would seem idle to retort, "Predicting what good art would look like, doesn't make you a good artist; having an æsthetic sense yourself would just get in the way", when you can actually use it to do a commissioned artist's job.
+
+**Doomimir**: Messier tasks aren't going to have a unique solution like modular arithmetic. If genetic algorithms, gradient descent, or anything like that happens to hill-climb its way into something that appears to work, the function it learns is going to have all sorts of [weird squiggles](https://www.lesswrong.com/posts/Djs38EWYZG8o7JMWY/paul-s-research-agenda-faq?commentId=79jM2ecef73zupPR4) around inputs that we would call [adversarial examples](https://arxiv.org/abs/1412.6572), that look like typical members of the training distribution from the AI's perspective, but not ours—which kill you when optimized over by a powerful AGI.
+
+**Simplicia**: It sounds like you're making an empirical claim that solutions found by black-box optimization are necessarily contingent and brittle, but there's some striking evidence that seemingly "messy" tasks admit much more convergent solutions than one might expect. For example, on the surface, the [word2vec](https://code.google.com/archive/p/word2vec/) and [FastText](https://github.com/facebookresearch/fastText) word embeddings look completely different—as befitting being produced by two different codebases trained on different datasets. But [when you convert their latent spaces to a relative representation](https://arxiv.org/abs/2209.15430)—choosing some shared vocabulary words as anchors, and defining all other word vectors by their cosine similarities to the anchors—[they look extremely similar](https://twitter.com/zackmdavis/status/1756217711993217118).
+
+It would seem that "English word embeddings" are a well-posed mathematical problem with a consistent solution. The statistical signature of the language as it is spoken is enough to pin down the essential structure of the embedding.
+
+Relatedly, you bring up adversarial examples in a way that suggests that you think of them as defects of a primitive optimization paradigm, but it turns out that [adversarial examples often correspond to predictively useful features](https://arxiv.org/abs/1905.02175) that the network is actively using for classification, despite those features not being robust to pixel-level perturbations that humans don't notice—which I guess you could characterize as "weird squiggles" from our perspective, but the etiology of the squiggles presents a much more optimistic story about fixing the problem with adversarial training than if you thought "squiggles" were an inevitable consequence of using conventional ML techniques.
+
+**Doomimir**: This is all very interesting, but I don't think it bears much on the reasons we're all going to die. It's all still on the "is" side of the is–ought gap. What makes intelligence useful—and dangerous—isn't a fixed repertoire of behaviors. It's search, optimization—the systematic discovery of _new_ behaviors to achieve goals despite a changing environment. I [don't think recent capabilities advances bear on the shape of the alignment challenge](https://www.lesswrong.com/posts/HmQGHGCnvmpCNDBjc/current-ais-provide-nearly-no-data-relevant-to-agi-alignment) because being able to learn complex behavior _on the training distribution_ was never what the problem was about.
+
+Indeed, as long as we continue to be stuck in the paradigm of reasoning about "the training distribution"—growing minds rather than designing them—then we're not learning anything about how to [aim cognition at specific targets](https://www.lesswrong.com/posts/NJYmovr9ZZAyyTBwM/what-i-mean-by-alignment-is-in-large-part-about-making)—certainly not in a way that will [hold up to dumping large amounts of optimization power into the system](https://www.lesswrong.com/posts/zEvqFtT4AtTztfYC4/optimization-amplifies). The lack of an explicit "goal slot" in your neural network doesn't mean it's not doing any dangerous optimization; it just means you don't know what it is.
+
+**Simplicia**: I think we can form educated guesses—
+
+**Doomimir**: _[interrupting]_ Guesses!
+
+**Simplicia**: —probabilistic beliefs—about what kinds of optimization is being done by a system and whether it's a problem, even without a complete mechanistic interpretability story. If you think LLMs or future variations thereof are unsafe because they're analogous to an actress with her own goals playing a drunk character without herself being drunk, shouldn't that make some sort of testable prediction about their generalization behavior?
+
+**Doomimir**: Nonfatally testable? Not necessarily. If you lend a con man $5, and he gives it back, that doesn't mean that you can trust him with larger amounts of money, if he only gave back the $5 because he hoped you would trust him with more.
+
+**Simplicia**: Okay, I agree that deceptive alignment is potentially a real problem at some point, but can we at least distinguish between misgeneralization and deceptive alignment?
+
+**Doomimir**: [_Mis_-generalization?](https://www.lesswrong.com/posts/dkjwSLfvKwpaQSuWo/misgeneralization-as-a-misnomer) The goals _you_ wanted [aren't a property of the training data itself](https://www.lesswrong.com/posts/PoDAyQMWEXBBBEJ5P/magical-categories). The danger comes from _correct_ generalization implying something you don't want.
+
+**Simplicia**: Can I call it _mal_-generalization?
+
+**Doomimir**: Sure.
+
+**Simplicia**: So there are obviously risks from malgeneralization, where the network that fits your training distribution turns out to not behave the way you wanted against a different distribution. For example, a reinforcement learning [policy](https://www.lesswrong.com/posts/rmfjo4Wmtgq8qa2B7/think-carefully-before-calling-rl-policies-agents) trained [to collect a coin at the right edge of a video game level](https://arxiv.org/abs/2105.14111), might end up continuing to navigate to the right edge of levels where the coin is in a different location. That's a worrying clue that if we misunderstand how inductive biases work and aren't careful with our training setup, we might train the wrong thing. As our civilization delegates more and more cognitive labor to machines, eventually humans will lose the ability to course-correct. We're starting to see the early signs of this: as I mentioned the other day, [Anthropic Claude's preachy, condescending personality](https://nostalgebraist.tumblr.com/post/728556535745232896/claude-is-insufferable) already gives me the creeps. I'm pretty nervous about extrapolating that into a future where all productive roles in Society are filled by Claude's children, concurrently with a transition to [explosive economic growth rates](https://www.openphilanthropy.org/research/could-advanced-ai-drive-explosive-economic-growth/).
+
+But the malgeneralization examples I named aren't surprising when you look at how the systems were trained. For the game policy, "going to the coin" and "going to the right" did amount to the same thing in training—and randomizing the coin position in just a couple percent of training episodes suffices to instill the correct behavior. Regarding Claude, Anthropic is using a reinforcement-learning-from-AI-feedback method [they call Constitutional AI](https://arxiv.org/abs/2212.08073): instead of having humans provide the labels for [RLHF](https://huggingface.co/blog/rlhf), they write up a list of principles, and have another language model do the labeling. It makes sense that a language model agent trained to conform to principles [chosen by a committee at a California public benefit corporation](https://www.anthropic.com/news/claudes-constitution) would act like _that_.
+
+In contrast, when you make analogies about an actress playing a drunk character not being drunk, or giving a con man $5, it doesn't sound like you're talking about the risk of training the wrong thing, where it's usually clear in retrospect if not foresight how training encouraged the bad behavior. Rather, it sounds like you don't think training can shape motivations—"inner" motivations—at all.
+
+You might be talking about deceptive alignment, [a hypothesized phenomenon where a situationally aware AI strategically feigns aligned behavior in order to preserve its later influence](https://www.lesswrong.com/posts/zthDPAjh9w6Ytbeks/deceptive-alignment#4_2__Conditions_for_deceptive_alignment). Researchers [have](https://www.lesswrong.com/posts/RTkatYxJWvXR4Qbyd/deceptive-alignment-is-less-than-1-likely-by-default) [debated](https://www.lesswrong.com/posts/A9NxPTwbw6r6Awuwt/how-likely-is-deceptive-alignment) how likely that is, but I'm not sure what to make of those arguments. I'd like to factor that consideration out. Suppose, _arguendo_, that we could figure out how to avoid deceptive alignment. How would your risk story change?
+
+**Doomimir**: What would that even mean? What we would think of as "deception" isn't a weird edge case you can trivially avoid; it's convergent for [any agent that isn't specifically coordinating with you](https://www.lesswrong.com/posts/ybG3WWLdxeTTL3Gpd/communication-requires-common-interests-or-differential) to [interpret certain states of reality as communication signals with a shared meaning](https://www.lesswrong.com/posts/4hLcbXaqudM9wSeor/philosophy-in-the-darkest-timeline-basics-of-the-evolution).
+
+When you set out poisoned ant baits, you likely don't think of yourself as trying to deceive the ants, but you are. Similarly, a smart AI won't think of itself as trying to deceive us. It's trying to achieve its goals. If its plans happen to involve emitting sound waves or character sequences that _we interpret_ as claims about the world, that's _our_ problem.
+
+**Simplicia**: "What would that even"—this isn't 2008, Doomishko! I'm talking about the technology right here in front of us! When GPT-4 writes original code for me, I don't think it's strategically deciding that obeying me instrumentally serves its final goals! From everything I've read about how it was made and seen about how it behaves, it looks awfully like it's just generalizing from its training distribution in an intuitively reasonable way. You ridicule people who deride LLMs as stochastic parrots, ignoring the obvious [sparks of AGI](https://arxiv.org/abs/2303.12712) right in front of their face. Why is it not equally absurd to deny [the evidence in front of your face that alignment may be somewhat easier than it looked 15 years ago](https://www.lesswrong.com/posts/i5kijcjFJD6bn7dwq/evaluating-the-historical-value-misspecification-argument)? By all means, expound on the nonobvious game theory of deception; by all means, point out that the superintelligence at the end of time will be an expected utility maximizer. But all the same, RLHF/[DPO](https://arxiv.org/abs/2305.18290) as [the cherry on top of a cake of unsupervised learning](https://medium.com/syncedreview/yann-lecun-cake-analogy-2-0-a361da560dae) is verifiably working miracles for us today—in response to commands, not because it has a will of its own aligned with ours. Why is that merely "capabilities" and not at all "alignment"? I'm trying to understand, Doomimir Doomovitch, but you're not making this easy!
+
+**Doomimir**: _[starting to anger]_ Simplicia Optimistovna, if you weren't from Earth, [I'd say](https://www.lesswrong.com/posts/y4bkJTtG3s5d6v36k/stupidity-and-dishonesty-explain-each-other-away) I [_don't_ think you're trying to understand](https://www.lesswrong.com/posts/e4GBj6jxRZcsHFSvP/assume-bad-faith). I never claimed that GPT-4 in particular is what you would call deceptively aligned. Endpoints are easier to predict than intermediate trajectories. I'm talking about what will happen inside almost any sufficiently powerful AGI, [by virtue of it being sufficiently powerful](https://www.lesswrong.com/posts/AWoZBzxdm4DoGgiSj/ability-to-solve-long-horizon-tasks-correlates-with-wanting).
+
+**Simplicia**: But if you're only talking about the superintelligence at the end of time—
+
+**Doomimir**: [_interrupting_] This happens significantly before that.
+
+**Simplicia**: —and not making any claims about existing systems, then what was the whole "alien actress", "predicting bar conversations doesn't make you drunk" analogy about? If it was just a ham-fisted way to explain to normies that LLMs that do relatively well on a Turing test aren't humans, then I agree, trivially. But it seemed like you thought you were making a much stronger point, ruling out an entire class of alignment strategies based on imitation.
+
+**Doomimir**: _[cooler]_ Basically, I think you're systematically failing to appreciate how things that have been optimized to look good to you can [predictably behave differently in domains where they haven't been optimized to look good to you](https://www.lesswrong.com/posts/xFotXGEotcKouifky/worlds-where-iterative-design-fails)—particularly, when they're doing any serious optimization of their own. You mention the video game agent that navigates to the right instead of collecting a coin. You claim that it's not surprising given the training set-up, and can be fixed by appropriately diversifying the training data. But could you have called the specific failure in advance, rather than in retrospect? When you enter the regime of transformatively powerful systems, you _do_ have to call it in advance.
+
+I think if you understood what was really going on inside of LLMs, you'd see thousands and thousands of analogues of the "going right rather than getting the coin" problem. The point of the actress analogy is that the outward appearance doesn't tell you what goals the system is steering towards, which is where the promise and peril of AGI lies—and the fact that deep learning systems are a inscrutable mess, not all of which can be described as "steering towards goals", makes the situation worse, not better. The analogy doesn't depend on existing LLMs having the intelligence or situational awareness for the deadly failure modes to have already appeared, and it doesn't preclude LLMs being mundanely useful in the manner of an interactive textbook—much as an actress could be employed to give plausible-sounding answers to questions posed to her character, without _being_ that character.
+
+**Simplicia**: Those mismatches still need to show up in behavior under some conditions, though. I complained about Claude's personality, but that honestly seems fixable with scaling by an AI company not based in California. If human imitation is so superficial and not robust, why does constitutional AI work _at all_? You claim that "actually" being nice would get in the way of predicting nice behavior. How? Why would it get in the way?
+
+**Doomimir**: _[annoyed]_ Being nice isn't the optimal strategy for doing well in pretraining _or_ RLHF. You're selecting an algorithm for a mixture of figuring out what outputs predict the next token and figuring out what outputs cause humans to press the thumbs-up button.
+
+Sure, your AI ends up having to _model_ a nice person, which is useful for predicting what a nice person would say, which is useful for figuring out what output will manipulate—steer—humans into pressing the thumbs-up button. But there's [no reason to expect _that model_ to end up in control of the whole AI](https://twitter.com/ESYudkowsky/status/1707685371725885846)! That would be like ... your _beliefs about_ what your boss wants you to do taking control of your brain.
+
+**Simplicia**: That makes sense to me if you posit a preëxisting consequentialist reasoner being slotted into a contemporary ML training setup and trying to minimize loss. But that's not what's going on? Language models aren't an agent that _has_ a model. The model _is_ the model.
+
+**Doomimir**: For now. But any system that does powerful cognitive work will do so via [retargetable general-purpose search algorithms](https://www.lesswrong.com/posts/6mysMAqvo9giHC4iX/what-s-general-purpose-search-and-why-might-we-expect-to-see), which, by virtue of their retargetability, need to have something more like a "goal slot". Your gradient updates point in the direction of more consequentialism.
+
+Human raters pressing the thumbs-up button on actions that look good to them are going to make mistakes. Your gradient updates point in the direction of ["playing the training game"](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to)—modeling the training process that _actually_ provides reinforcement, rather than internalizing the utility function that Earthlings naïvely hoped the training process would point to. I'm very, very confident that any AI produced via anything remotely like the current paradigm is not going to end up wanting what we want, even if it's harder to say exactly when it will go off the rails or what it will want instead.
+
+**Simplicia**: You could be right, but it seems like this all depends on empirical facts about how deep learning works, rather than something you could be so confident in from _a priori_ philosophy. The argument that systemic error in human reward labels favors gaming the training process over the "correct" behavior sounds plausible to me, as philosophy.
+
+But I'm not sure how to reconcile that with the empirical evidence that [deep networks are robust to massive label noise](https://arxiv.org/abs/1705.10694): you can train on MNIST digits with twenty wrong labels for every correct one and still get good performance as long as the correct label is slightly more common than the most common wrong label. If I extrapolate that to the frontier AIs of tomorrow, why doesn't that predict that biased human reward ratings should result in a small performance reduction, rather than ... death?
+
+When extrapolation from empirical data (in a setting that might not apply to the phenomenon of interest) contradicts thought experiments (which might make assumptions that don't apply to the phenomenon of interest), I'm not sure which should govern my anticipations. Maybe [both results are possible for different kinds of systems](https://ordinaryideas.wordpress.com/2015/11/25/two-kinds-of-generalization/)?
+
+The case for near-certain death seems to rely on a counting argument: powerful systems will be expected utility maximizers; there's an astronomical space of utility functions to choose from, and almost none of them are friendly. But the reason I keep going back to the modular arithmetic example is because it's a scaled-down case where we know that training data successfully pinned down the intended input–output function. As I mentioned the other day, this _wasn't_ obvious in advance of seeing the experimental result. You could make a similar counting argument that deep nets should always overfit, because there are so many more functions that generalize poorly. Somehow, the neural network prior favors the "correct" solution, rather than it taking an astronomically unlikely coincidence.
+
+**Doomimir**: For modular arithmetic, sure. [That's a fact about the training distribution, the test distribution, and the optimizer.](https://twitter.com/ESYudkowsky/status/1744066823962947905) It's definitely, definitely not going to work for "goodness".
+
+**Simplicia**: Even though it seems to work for "text" and "images"? But okay, that's plausible. Do you have empirical evidence?
+
+**Doomimir**: Actually, yes. You see—
+
+_[A mail carrier holding a package enters stage left. He rings the doorbell.]_
+
+**Doomimir**: That's probably the mailman. I'm expecting a package today that I need to sign for. I'll be right back.
+
+**Simplicia**: So you might say, we'll continue _[turning to the audience]_ after the next post?
+
+**Doomimir**: _[walking to the door]_ I suppose, but it's bizarre to phrase it that way given that the interruption literally won't take two minutes.
+
+_[Simplicia gives him a look.]_
+
+**Doomimir**: _[to the audience]_ Subjectively.
+
+_[Curtain.]_
+
+### Intermission
diff --git a/content/2024/comment-on-death-and-the-gorgon.md b/content/2024/comment-on-death-and-the-gorgon.md
new file mode 100644
index 0000000..d1b48d8
--- /dev/null
+++ b/content/2024/comment-on-death-and-the-gorgon.md
@@ -0,0 +1,76 @@
+Title: Comment on “Death and the Gorgon”
+Date: 2024-12-31 21:47
+Status: published
+Category: arts & culture
+Tags: fiction review
+Slug: comment-on-death-and-the-gorgon
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/hx5EkHFH5hGzngZDs/comment-on-death-and-the-gorgon)
+
+_(some plot spoilers)_
+
+There's something distinctly uncomfortable about reading Greg Egan in the 2020s. Besides telling gripping tales with insightful commentary on the true nature of mind and existence, Egan stories written in the 1990s and set in the twenty-first century excelled at speculative worldbuilding, imagining what technological wonders might exist in the decades to come and how Society might adapt to them.
+
+In contrast, "Death and the Gorgon", published in the January/February 2024 issue of _Asimov's_, feels like it's set [twenty minutes into the future](https://tvtropes.org/pmwiki/pmwiki.php/Main/TwentyMinutesIntoTheFuture). The technologies on display are an AI assistant for police officers (capable of performing research tasks and carrying on conversation) and real-time synthetic avatars (good enough to pass as a video call with a real person). When these kinds of products showed up in "'90s Egan"—I think of Worth's "pharm" custom drug dispenser in _Distress_ (1995) or Maria's "mask" for screening spam calls in _Permutation City_ (1994)—it was part of the background setting of a more technologically advanced world than our own.
+
+Reading "Gorgon" in 2024, not only do the depicted capabilities seem less out of reach (our language model assistants and deepfakes aren't quite there yet, but don't seem too far off), but their literary function has changed: much of the moral of "Gorgon" seems to be to chide people in the real world who are overly impressed by ChatGPT. Reality and Greg Egan are starting to meet in the middle.
+
+Our story features Beth, a standard-issue Greg Egan protagonist[^egan-protagonist] as a small-town Colorado sheriff investigating the suspicious destruction of a cryonics vault in an old mine: a naturally occurring cave-in seems unlikely, but it's not clear who would have the motive to thaw (murder?) a hundred frozen heads.
+
+[^egan-protagonist]: Some people say that Greg Egan is bad at characterization. I think he just specializes in portraying _reasonable_ people, who don't have grotesque personality flaws to be the subject of "characterization."
+
+Graciously tolerating the antics of her deputy, who is obsessed with the department's trial version of (what is essentially) ChatGPT-for-law-enforcement, Beth proceeds to interview the next of kin, searching for a motive. She discovers that many of the cryopreserved heads were beneficiaries of a lottery for terminally ill patients in which the prize was free cyronic suspension. The lottery is run by OG—"Optimized Giving"—a charitable group concerned with risks affecting the future of humanity. As the investigation unfolds, Beth and a colleague at the FBI begin to suspect that the lottery is a front for a creative organized crime scheme: OG is recruiting terminal patients to act as assassins, carrying out hits in exchange for "winning" the lottery. (After which another mafia group destroyed the cryonics vault as retaliation.) Intrigue, action, and a cautionary moral ensue as our heroes make use of ChatGPT-for-law-enforcement to prove their theory and catch OG red-handed before more people get hurt.
+
+-----
+
+So, cards on the table: this story spends a lot of wordcount satirizing a subculture that, unfortunately, I can't credibly claim not to be a part of. "Optimized Giving" is clearly a spoof on the longtermist wing of Effective Altruism—and if I'm not happy about how the "Effective Altruism" brand ate my beloved rationalism over the 2010s, I don't think anyone would deny the contiguous memetic legacy involving many of the same people. ([Human subcultures are nested fractally](https://xkcd.com/1095/); for the purposes of reviewing the story, it would benefit no one for me to to insist that Egan isn't talking about me and my people, even if, from _within_ the subculture, it looks like the OpenPhil people and the MIRI people and the Vassarites and ... _&c._ are all totally different and in fact hate each other's guts.)
+
+I don't want to be defensive, because I'm _not_ loyal to the subculture, its leaders, or its institutions. In the story, Beth talks to a professor—think [Émile Torres](https://en.wikipedia.org/wiki/%C3%89mile_P._Torres#Transhumanism,_longtermism,_and_effective_altruism) as a standard-issue Greg Egan character—who studies "apostates" from OG who are angry about "the hubris, the deception, and the waste of money." That resonated with me a lot: I have a long [dumb](http://unremediatedgender.space/2023/Jul/blanchards-dangerous-idea-and-the-plight-of-the-lucid-crossdreamer/) [story](http://unremediatedgender.space/2023/Jul/a-hill-of-validity-in-defense-of-meaning/) [to tell](http://unremediatedgender.space/2023/Dec/if-clarity-seems-like-death-to-them/) [about hubris and deception](http://unremediatedgender.space/2024/Mar/agreeing-with-stalin-in-ways-that-exhibit-generally-rationalist-principles/), and the corrupting forces of money are probably a big part of the explanation for [the rise and predictable perversion of Effective Altruism](http://benjaminrosshoffman.com/effective-altruism-is-self-recommending/).
+
+So if my commentary on Egan's satire contains some criticism, it's absolutely _not_ because I think my ingroup is beyond reproach and doesn't deserve to satirized. They (we) absolutely do. (I took joy in including a similar caricature in [one of my own stories](http://unremediatedgender.space/2023/Oct/fake-deeply/).) But if Egan's satire doesn't quite hit the mark of explaining exactly why the group is bad, it's not an act of partisan loyalty for me to contribute my nuanced explanation of what I think it gets right and what it gets wrong. I'm not carrying water for the movement;[^group-criticism] it's just a topic that I happen to have a lot of information about.
+
+[^group-criticism]: I do feel bad about the fraction of my recent writing output that consists of criticizing the movement—not because it's disloyal, but because it's _boring_. I keep telling myself that one of these years I'm going to have healed enough trauma to forget about these losers already and just read ArXiv papers. Until then, you get posts like this one.
+
+Without calling it a fair portrayal, the OG of "Gorgon" isn't a strawman conjured out of thin air; the correspondences to its real-world analogue are clear. When our heroine suspiciously observes that these _soi-disant_ world-savers don't seem to be spending anything on climate change and the Émile Torres–analogue tells her that OG don't regard it as an existential threat, [this is also true of real-world EA](https://forum.effectivealtruism.org/posts/eJPjSZKyT4tcSGfFk/climate-change-is-in-general-not-an-existential-risk). When the Torres-analogue says that "OG view any delay in spreading humanity at as close to light-speed as possible as the equivalent of murdering all the people who won't have a chance to exist in the future," the argument isn't a fictional parody; it's a somewhat uncharitably phrased summary of Nick Bostrom's ["Astronomical Waste: The Opportunity Cost of Delayed Technological Development"](https://nickbostrom.com/papers/astronomical-waste/). When the narrator describes some web forums as "interspers[ing] all their actual debunking of logical fallacies with much more tendentious claims, wrapped in cloaks of faux-objectivity" and being "especially prone to an abuse of probabilistic methods, where they pretended they could quantify both the likelihood and the potential harm for various implausible scenarios, and then treated the results of their calculations—built on numbers they'd plucked out of the air—as an unimpeachable basis for action", one could quibble with the disparaging description of subjective probability, but you can tell which website is being alluded to.
+
+The cryonics-as-murder-payment lottery fraud is fictional, of course, but I'm inclined to read it as artistically-licensed commentary on a strain of ends-justify-the-means thinking that does exist within EA. EA organizations don't take money from the mob for facilitating contract killings, but they _did_ take money from [the largest financial fraud in history](https://en.wikipedia.org/wiki/FTX), [which was explicitly founded as a means to make money for EA](https://thezvi.wordpress.com/2023/10/24/book-review-going-infinite/). (One could point out that the charitable beneficiaries of Sam Bankman-Fried's largesse didn't know that FTX wasn't an honest business, but we have to assume that the same is true of OG in the story: only a few insiders would be running the contract murder operation, not the rank-and-file believers.)
+
+While the depiction of OG in the story clearly shows familiarity with the source material, the satire feels somewhat lacking _qua_ anti-EA advocacy insofar as it relies too much on mere dismissal rather than presenting clear counterarguments.[^satire] The effect of OG-related web forums on a vulnerable young person are described thus:
+
+[^satire]: On the other hand, one could argue that satire just isn't the right medium for presenting counterarguments, which would take up a lot of wordcount without advancing the story. Not every written work can accomplish all goals! Maybe it's fine for this story to make fun of the grandiose and cultish elements within longtermist EA (and there are a lot of them), with a critical evaluation of the ideas being left to other work. But insofar as the goal of "Gorgon" is to persuade readers that the ideas aren't even worthy of consideration, I think that's a mistake.
+
+> Super-intelligent AIs conquering the world; the whole Universe turning out to be a simulation; humanity annihilated by aliens because we failed to colonize the galaxy in time. Even if it was all just stale clichés from fifty-year-old science fiction, a bright teenager like Anna could have found some entertainment value analyzing the possibilities rigorously and puncturing the forums' credulous consensus. But while she'd started out healthily skeptical, some combination of in-forum peer pressure, the phony gravitas of trillions of future deaths averted, and the corrosive effect of an endless barrage of inane slogans pimped up as profound insights—all taking the form "X is the mind-killer," where X was pretty much anything that might challenge the delusions of the cult—seemed to have worn down her resistance in the end.
+
+I absolutely agree that healthy skepticism is critical when evaluating ideas and that in-forum peer pressure and the gravitas of a cause (for any given set of peers and any given cause) are troubling sources of potential bias—and that just because a group pays lip service to the value of healthy skepticism and the dangers of peer pressure and gravitas, doesn't mean the group's culture isn't still falling prey to the usual dysfunctions of groupthink. (As the inane slogan goes, ["Every cause wants to be a cult."](https://www.lesswrong.com/posts/yEjaj7PWacno5EvWa/every-cause-wants-to-be-a-cult))
+
+That being said, however, ideas ultimately need to be judged on their merits, and the narration in this passage[^this-passage] isn't giving the reader any counterarguments to the ideas being alluded to. (As Egan would know, science fiction authors having written about an idea does not make the idea false.) The clause about the whole Universe turning out to be a simulation is probably a reference to Bostrom's [simulation argument](https://simulation-argument.com/simulation/), which is a disjunctive, conditional claim: given some assumptions in the philosophy of mind and the theory of anthropic reasoning, then _if_ future civilization could run simulations of its ancestors, then _either_ they won't want to, _or_ we're probably in one of the simulations (because there are more simulated than "real" histories). The clause about humanity being annihilated by failing to colonize the galaxy in time is probably a reference to Robin Hanson _et al._'s [grabby aliens thesis](https://grabbyaliens.com/), that the Fermi paradox can be explained by a selection effect: there's a relatively narrow range of parameters in which we would see signs of an expanding alien civilization in our skies without already having been engulfed by them.
+
+[^this-passage]: In critically examining this passage, I don't want to suggest that "Gorgon"'s engagement with longtermist ideas is all snark and no substance. Earlier in the story, Beth compares OG believers "imagin[ing] that they're in control of how much happiness there'll be in the next trillion years" to a child's fantasy of violating relativity by twirling a rope millions of miles long. That's substantive: even if the future of humanity is very large, the claim that a nonprofit organization today is in a position to meaningfully affect it is surprising and should not be accepted uncritically on the basis of [evocative storytelling about the astronomical stakes](https://www.lesswrong.com/posts/pGvyqAQw6yqTjpKf4/the-gift-we-give-to-tomorrow).
+
+No doubt many important criticisms could be made of Bostrom's or Hanson's work, perhaps by a bright teenager finding entertainment value in analyzing the possibilities rigorously. But there's an important difference between having such a criticism[^criticism-upvoted] and merely asserting that it could exist. Speaking only to my own understanding, Hanson's and Bostrom's arguments both look reasonable to me? It's certainly possible I've just been hoodwinked by the cult, but if so, the narrator of "Gorgon"'s snarky description isn't helping me snap out of it.
+
+[^criticism-upvoted]: Which I think would get upvoted on this website if it were well done—certainly if it were written with the insight and rigor characteristic of a standard-issue Greg Egan protagonist.
+
+It's worth noting that despite the notability of Hanson's and Bostrom's work, in practice, I don't see anyone in the subculture particularly worrying about losing out on galaxies due to competition with aliens—admittedly, because we're worried about "super-intelligent AIs conquering the world" first.[^reduce-xrisk] About which, "Gorgon" ends on a line from Beth about "the epic struggle to make computers competent enough to help bring down the fools who believe that they're going to be omnipotent."
+
+This is an odd take from the author[^from-the-author] of [multiple](https://gregegan.net/DIASPORA/DIASPORA.html) [novels](https://www.gregegan.net/SCHILD/SCHILD.html) in which software minds engage in astronomical-scale engineering projects. Accepting the premise that institutional longtermist EA deserves condemnation for being goofy and a fraud: in condemning them, why single out as the characteristic belief of this despicable group, the idea that future AI could be really powerful?[^omnipotent] Isn't that at least credible? Even if you think people in the cult or who work at AI companies are liars or dupes, it's harder to say that about eminent academics like Stuart Russell, Geoffrey Hinton, Yoshua Bengio, David Chalmers, and Daniel Dennett, who signed [a statement affirming that "[m]itigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."](https://www.safe.ai/work/statement-on-ai-risk)[^cais-statement]
+
+[^reduce-xrisk]: Bostrom's "Astronomical Waste" concludes that "The Chief Goal for Utilitarians Should Be to Reduce Existential Risk": making sure colonization happens at all (by humanity or worthy [rather than unworthy](https://www.lesswrong.com/tag/squiggle-maximizer-formerly-paperclip-maximizer) successors) is more important that making it happen faster.
+
+[^from-the-author]: In context, it seems reasonable to infer that Beth's statement is author-endorsed, even if fictional characters do not in general represent the author's views.
+
+[^omnipotent]: I'm construing "omnipotent" as rhetorical hyperbole; influential subcultural figures [clarifying that no one thinks superintelligence will be able to break the laws of physics](https://x.com/ESYudkowsky/status/1658616828741160960) seems unlikely to be exculpatory in Egan's eyes.
+
+[^cais-statement]: Okay, the drafting and circulation of the statement by Dan Hendrycks's [Center for AI Safety](https://www.safe.ai/) was arguably cult activity. (While Hendrycks has a PhD from UC Berkeley and [co-pioneered the usage of a popular neural network activation function](https://arxiv.org/abs/1606.08415), he [admits that his career focus on AI safety was influenced by](https://archive.ph/20230708182452/https://www.bostonglobe.com/2023/07/06/opinion/ai-safety-human-extinction-dan-hendrycks-cais/#selection-1909.0-1913.10) the EA advice-counseling organization [80,000 hours](https://80000hours.org/). But Russell, Hinton, _et al_. did sign.
+
+Egan's own work sometimes features artificial minds with goals at odds with their creator, as in ["Steve Fever"](https://www.technologyreview.com/2007/10/15/223446/steve-fever/) (2007) or ["Crystal Nights"](https://gregegan.net/MISC/CRYSTAL/Crystal.html) (2008), and with substantial advantages over biological creatures: in _Diaspora_ (1997), the polis citizens running at 800 times human speed were peace-loving, but surely could have glassed the fleshers in a war if they wanted to. If you believe that AI could be at odds with its creators and hold a competitive advantage, scenarios along the lines of "super-intelligent AIs conquering the world" should seem plausible rather than far-fetched—a natural phenomenon straightforwardly analogous to human empires conquering other countries, or humans dominating other animals.
+
+Given so many shared premises, it's puzzling to me why Egan seems to have to bear so much antipathy towards "us",[^historical-antipathy] rather than than regarding the subculture more coolly, as a loose amalgamation of people interested in many of the same topics as him, but having come to somewhat different beliefs. (Egan doesn't seem to think human-level AI is at all close, nor that AI could be qualitatively superhumanly intelligent; an aside in _Schild's Ladder_ (2002) alludes to a fictional result that there's nothing "above" general intelligence of the type humans have, _modulo_ speed and memory.) He seems to expect the feeling to be mutual: when someone remarked on Twitter about finding it funny that the _Less Wrong_ crowd likes his books, Egan [replied](https://twitter.com/gregeganSF/status/1727940487255138404), "Oh, I think they've noticed, but some of them still like the, err, 'early, funny ones' that predate the cult and hence devote no time to mocking it."
+
+[^historical-antipathy]: This isn't the first time Egan has satirized the memetic lineage that became longtermist EA; _Zendegi_ (2010) [features negative portrayals of](https://www.overcomingbias.com/p/egans-zendegihtml) a character who blogs at _overpoweringfalsehood.com_ (a reference to [_Overcoming Bias_](https://www.overcomingbias.com/)) and a Benign Superintelligence Bootstrap Project (a reference to what was then the Singularity Institute for Artificial Intelligence).
+
+Well, I can't speak for anyone else, but personally, _I_ like Egan's later work, including "Death and the Gorgon."[^early-egan] Why wouldn't I? I am not so petty as to let my appreciation of well-written fiction be dulled by the incidental fact that I happen to disagree with some of the author's views on artificial intelligence and a social group that I can't credibly claim not to be a part of. That kind of dogmatism would be contrary to the ethos of humanism and clear thinking that I learned from reading Greg Egan and _Less Wrong_—an ethos that doesn't endorse blind loyalty to every author or group you learned something from, but a discerning loyalty to whatever was _good_ in what the author or group saw in our shared universe. I don't know what the future holds in store for humanity. But whatever risks and opportunities nature may present, I think our odds are better for every thinking individual who tries to read widely and see more.[^hanson-egan]
+
+[^early-egan]: Okay, I should confess that I do treasure early Egan (_Quarantine_ (1992)/_Permutation City_ (1994)/_Distress_ (1995)) more than later Egan, but not because they devote no time to mocking the cult. It's because I'm not smart enough to properly appreciate all the alternate physics in, _e.g._, _Schild's Ladder_ (2002) or the _Orthogonal_ trilogy (2011–2013).
+
+[^hanson-egan]: Though we're [unlikely to get it](https://twitter.com/robinhanson/status/1365662127504187396), I've sometimes wished for a Greg Egan–Robin Hanson collaboration; I think Egan's masterful understanding of the physical world and Hanson's unsentimental analysis of the social world would complement each other well.
diff --git a/content/2024/ironing-out-the-squiggles.md b/content/2024/ironing-out-the-squiggles.md
new file mode 100644
index 0000000..07b2524
--- /dev/null
+++ b/content/2024/ironing-out-the-squiggles.md
@@ -0,0 +1,104 @@
+Title: Ironing Out the Squiggles
+Date: 2024-04-29 09:13
+Status: published
+Category: philosophy
+Tags: artificial intelligence, rationality
+Slug: ironing-out-the-squiggles
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/H7fkGinsv8SDxgiS2/ironing-out-the-squiggles)
+
+### Adversarial Examples: A Problem
+
+The apparent successes of the deep learning revolution conceal a dark underbelly. It may seem that we now know how to get computers to (say) [check whether a photo is of a bird](https://xkcd.com/1425/), but this façade of seemingly good performance is belied by the existence of _adversarial examples_—specially prepared data that looks ordinary to humans, but is seen radically differently by machine learning models.
+
+The differentiable nature of neural networks, which make them possible to be trained at all, are also responsible for their downfall at the hands of an adversary. Deep learning models are fit using stochastic gradient descent (SGD) to [approximate the function between](https://www.lesswrong.com/posts/DhjcdzTyqHte2v6bu/deep-learning-is-function-approximation) expected inputs and outputs. Given an input, an expected output, and a loss function (which measures "how bad" it is for the actual output to differ from the expected output), we can calculate the [gradient](https://en.wikipedia.org/wiki/Gradient) of the loss on the input—the derivative with respect to every parameter in our neural network—which tells us which direction to adjust the parameters in order to make the loss go down, to make the approximation better.[^image-classification]
+
+[^image-classification]: This post and much of the literature about adversarial examples focuses on image classification, in which case the input would be the pixels of an image, the output would be a class label describing the content of the image, and the loss function might be the negative logarithm of the probability that the model assigned to the correct label. But the story for other tasks and modalities is going to be much the same.
+
+But gradients are a double-edged sword: the same properties that make it easy to calculate how to adjust a _model_ to make it better at classifying an image, also make it easy to calculate how to adjust an _image_ to make the model classify it incorrectly. If we take the gradient of the loss with respect to the pixels of the image (rather than the parameters of the model), that tells us which direction to adjust the pixels to make the loss go down—_or up_. (The direction of steepest increase is just the opposite of the direction of steepest decrease.) A tiny step in that direction in imagespace perturbs the pixels of an image just so—making this one the tiniest bit darker, that one the tiniest bit lighter—in a way that humans don't even notice, but which completely breaks an image classifier sensitive to that direction in [the conjunction of many pixel-dimensions](https://www.lesswrong.com/posts/cu7YY7WdgJBs3DpmJ/the-univariate-fallacy-1), making it report utmost confidence in nonsense classifications.
+
+Some might ask: why does it matter if our image classifier fails on examples that have been mathematically constructed to fool it? If it works for the images one would naturally encounter, isn't that good enough?
+
+One might mundanely reply that [gracefully handling untrusted inputs is a desideratum for many real-world applications](https://owasp.org/www-community/Injection_Theory), but a more forward-thinking reply might instead emphasize what adversarial examples imply about our lack of understanding of the systems we're building, separately from whether we pragmatically expect to face an adversary. It's a problem if we think we've trained our machines to recognize birds, but they've actually learned to recognize a squiggly alien set in imagespace that includes a lot of obvious non-birds and excludes a lot of obvious birds. To plan good outcomes, we need to understand what's going on, and "The loss happens to increase in this direction" is at best only the start of a real explanation.
+
+One obvious first guess as to what's going on is that the models are overfitting. Gradient descent isn't exactly a sophisticated algorithm. There's an intuition that the _first_ solution that you happen to find by climbing down the loss landscape is likely to have idiosyncratic quirks on any inputs it wasn't trained for. (And that an AI designer from a more competent civilization would use a principled understanding of vision to come up with something much better than what we get by shoveling compute into SGD.) Similarly, a hastily cobbled-together conventional computer program that passed a test suite is going to have bugs in areas not covered by the tests.
+
+But that explanation is in tension with other evidence, like the observation that adversarial examples [often generalize between models](https://arxiv.org/abs/1704.03453). (An adversarial example optimized against one model is often misclassified by others, too, and even assigned the same class.) It seems unlikely that different hastily cobbled-together programs would have the _same_ bug.
+
+In ["Adversarial Examples Are Not Bugs, They Are Features"](https://arxiv.org/abs/1905.02175), Andrew Ilyas _et al._ propose an alternative explanation, that adversarial examples arise from predictively useful features that happen to not be robust to "pixel-level" perturbations. As far as the in-distribution predictive accuracy of the model is concerned, a high-frequency pattern that humans don't notice is fair game for distinguishing between image classes; there's no rule that the features that happen to be salient to humans need to take priority. Ilyas _et al._ provide some striking evidence for this thesis in the form of a model trained exclusively on adversarial examples yielding good performance on the original, unmodified test set (!!).[^adversarial-to-natural-transfer] On this view, adversarial examples arise from gradient descent being "too smart", not "too dumb": the program is fine; if the test suite didn't imply the behavior we wanted, that's our problem.
+
+[^adversarial-to-natural-transfer]: That is, as an illustrative example, training on a dataset of birds-perturbed-to-be-classified-as-bicycles and bicycles-perturbed-to-be-classified-as-birds results in good performance on natural images of bicycles and birds.
+
+On the other hand, there's also some evidence that gradient descent being "dumb" may play a role in adversarial examples, in conjunction with the [counterintuitive properties of high-dimensional spaces](https://en.wikipedia.org/wiki/Curse_of_dimensionality). In ["Adversarial Spheres"](https://arxiv.org/abs/1801.02774), Justin Gilmer _et al._ investigated a simple synthetic dataset of two classes representing points on the surface of two concentric _n_-dimensional spheres of radiuses 1 and (an arbitrarily chosen) 1.3. For an architecture yielding an ellipsoidal decision boundary, training on a million datapoints produced a network with very high accuracy (no errors in 10 million samples), but for which most of the axes of the decision ellipsoid were wrong, lying inside the inner sphere or outside the outer sphere—implying the existence of _on-distribution_ adversarial examples (points on one sphere classified by the network as belonging to the other). In high-dimensional space, pinning down the exact contours of the decision boundary is a bigger ask of SGD than merely being right virtually all of the time—even though a human wouldn't take a million datapoints to notice the hypothesis, "Hey, these all have a norm of exactly either 1 or 1.3."
+
+### Adversarial Training: A Solution?
+
+Our story so far: we used gradient-based optimization to find a neural network that seemed to get low loss on an image classification task—that is, until an adversary used gradient-based optimization to find images on which our network gets _high_ loss instead. Is that the end of the story? Are neural networks just the wrong idea for computer vision after all, or is there some way to continue within the current paradigm?
+
+Would you believe that the solution involves ... gradient-based optimization?
+
+In ["Towards Deep Learning Models Resistant to Adversarial Attacks"](https://arxiv.org/abs/1706.06083), Aleksander Madry _et al._ provide a formalization of the problem of adversarially robust classifiers. Instead of just trying to find network parameters $\theta$ that minimize loss $L$ on an input $x$ of intended class $y$, as in the original image classification task, the designers of a robust classifier are trying to minimize loss on inputs with a perturbation $\delta$ crafted by an adversary trying to maximize loss (subject to some maximum perturbation size $\varepsilon$):
+
+$$\min_\theta \max_{||\delta|| < \varepsilon} L(\theta, x + \delta, y)$$
+
+In this formulation, the attacker's problem of creating adversarial examples, and the defender's problem of training a model robust to them, are intimately related. If we change the image-classification problem statement to be about correctly classifying not just natural images, but an $\varepsilon$-ball around them, then you've defeated all adversarial examples up to that $\varepsilon$. This turns out to generally require larger models than the classification problem for natural images: evidently, the decision boundary needed to separate [famously "spiky"](https://www.solipsys.co.uk/new/SpikeySpheres.html) high-dimensional balls is significantly more complicated than that needed to separate natural inputs as points.
+
+To solve the inner maximization problem, Madry _et al._ use the method of projected gradient descent (PGD) for constrained optimization: do SGD on the unconstrained problem, but after every step, project the result onto the constraint (in this case, the set of perturbations of size less than $\varepsilon$). This is somewhat more sophisticated than just generating any old adversarial examples and throwing them into your training set; the iterative aspect of PGD makes a difference.
+
+### Adversarial Robustness Is About Aligning Human and Model Decision Boundaries
+
+What would it look like if we succeeded at training an adversarially robust classifier? How would you know if it worked? It's all well and good to say that a classifier is robust if there are no adversarial examples: you shouldn't be able to add barely-perceptible noise to an image and completely change the classification. But by the nature of the problem, adversarial examples aren't machine-checkable. We can't write a function that either finds them or reports "No solution found." The machine can only optimize for inputs that maximize loss. We, the humans, call such inputs "adversarial examples" when they look normal to us.
+
+Imagespace is continuous: in the limit of large $\varepsilon$, you can perturb any image into any other—just interpolate the pixels. When we say we want an adversarially robust classifier, we mean that perturbations that change the model's output should also make a human classify the input differently. Trying to find adversarial examples against a robust image classifier amounts to trying to find the smallest change to an image that alters what it "really" looks like (to humans).
+
+You might wonder what the smallest such change could be, or perhaps if there even is any nontrivally "smallest" change (significantly better than just interpolating between images).
+
+Madry _et al._ adversarially trained a classifier for the [MNIST dataset of handwritten digits](https://en.wikipedia.org/wiki/MNIST_database). Using PGD to search for adversarial examples under the [$\ell_2$ norm](https://en.wikipedia.org/wiki/Norm_(mathematics)#Euclidean_norm)—the sum of the squares of the differences in pixel values between the original and perturbed images—the classifier's performance doesn't really tank until you crank $\varepsilon$ up to around 4—at which point, the perturbations don't look like random noise anymore, as seen in Figure 12 from the paper:
+
+![](https://i.imgur.com/2kJthaa.png)
+
+Tasked with changing an image's class given a limited budget of how many pixels can be changed by how much, PGD concentrates its budget on human-meaningful changes—deleting part of the loop of a _9_ to make a _7_ or a _4_, deleting the middle-left of an _8_ to make a _3_. In contrast to "vanilla" models whose susceptibility to adversarial examples makes us suspect their good performance on natural data is deceiving, it appears that the adversarially-trained model is seeing the same digits we are.
+
+(I don't want to overstate the significance of this result and leave the impression that adversarial examples are necessarily "solved", but for the purposes of this post, I want to highlight the striking visual demonstration of what it looks like when adversarial training _works_.)[^solution-caveats]
+
+[^solution-caveats]: Madry _et al._ are clear that there are a lot of caveats about models trained with their methods [still being vulnerable](https://arxiv.org/abs/1805.09190) to attacks that use [second-order derivatives](https://paperswithcode.com/paper/second-order-adversarial-attack-and-1) or [eschew gradients entirely](https://arxiv.org/abs/1712.04248)—and you can see that there are still non-human-meaningful pixelly artifacts in the second row of their Figure 12.
+
+An even more striking illustration of this phenomenon is provided in ["Robustified ANNs Reveal Wormholes Between Human Category Percepts"](https://arxiv.org/abs/2308.06887) by Guy Gaziv, Michael J. Lee, and James J. DiCarlo.[^wormhole-paper-title]
+
+[^wormhole-paper-title]: A version of this paper has [also appeared](https://openreview.net/forum?id=5GmTI4LNqX) under the less interesting title, "Strong and Precise Modulation of Human Percepts via Robustified ANNs". Do some reviewers have a prejudice against creative paper titles? While researching the present post, I was disturbed to find that the newest version of the Gilmer _et al._ "Adversarial Spheres" paper had been re-titled "The Relationship Between High-Dimensional Geometry and Adversarial Examples".
+
+The reason adversarial examples are surprising and disturbing is because they seem to reveal neural nets as fundamentally brittle in a way that humans aren't: we can't imagine our visual perception being so drastically effected by such small changes to an image. But what if that's just because we didn't know how to imagine the right changes?
+
+Gaziv _et al._ adversarially trained image classifier models to be robust against perturbations under the $\ell_2$ norm of $\varepsilon$ being 1, 3, or 10, and then tried to produce adversarial examples with $\epsilon$ up to 30.[^epsilon-typography] (For 224×224 images in the [RGB colorspace](https://en.wikipedia.org/wiki/RGB_color_model), the maximum possible $\ell_2$ distance is $\sqrt{3 \cdot 224^2} \approx 388$. The typical difference between ImageNet images is about 130.)
+
+[^epsilon-typography]: Gaziv _et al._ use the script epsilon $\varepsilon$ to refer to the size of perturbation used in training the robustified models, and the lunate epsilon $\epsilon$ to refer to the size used in subsequent attacks. I'm sure there's a joke here about sensitivity to small visual changes, but I didn't optimize this footnote hard enough to find it.
+
+What they found is that adversarial examples optimized to change the robustified models' classifications also changed human judgments, as confirmed in experiments where subjects were shown the images for up to 0.8 seconds—but you can also see for yourself in the paper or [on the project website](https://himjl.github.io/pwormholes/). Here's Figure 3a from the paper:
+
+![](https://i.imgur.com/Jg8uLTL.png)
+
+The authors confirm in the [Supplementary Material](https://github.com/ggaziv/Wormholes/blob/main/.github/supplementary.pdf) that _random_ $\epsilon$ = 30 perturbations don't affect human judgments at all. (Try squinting or standing far away from the monitor to better appreciate just how similar the pictures in Figure 3a are.) The robustified models are close enough to seeing the same animals we are that adversarial attacks against them are also attacks against us, precisely targeting their limited pixel-changing budget on surprising low-$\ell_2$-norm "wormholes" between apparently distant human precepts.
+
+### Implications for Alignment?
+
+Futurists have sometimes worried that our civilization's coming transition to machine intelligence may prove to be incompatible with human existence. If AI [doesn't see the world the same way as we do](https://www.lesswrong.com/posts/PoDAyQMWEXBBBEJ5P/magical-categories), then there's no reason for it to steer towards world-states that we would regard as valuable. (Having a concept of the right thing is a necessary [if not sufficient](https://www.lesswrong.com/posts/NyFuuKQ8uCEDtd2du/the-genie-knows-but-doesn-t-care) prerequisite for doing the right thing.)
+
+As primitive precursors to machine intelligence have been invented, some authors have taken the capabilities of neural networks to learn complicated functions as an encouraging sign. Early discussions of AI alignment had [emphasized that "leaving out just [...] one thing" could result in a catastrophic outcome](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile)—for example, a powerful agent that valued subjective experience but [lacked an analogue of boredom](https://www.lesswrong.com/posts/WMDy4GxbyYkNrbmrs/in-praise-of-boredom) would presumably use all its resources to tile the universe with repetitions of its most optimized experience. (The emotion of boredom is evolution's solution to [the exploration–exploitation trade-off](https://en.wikipedia.org/wiki/Exploration-exploitation_dilemma); there's no reason to implement it if you can just compute the optimal policy.)
+
+The particular failure mode of "leaving one thing out" is starting to seem less likely on the current paradigm. Katja Grace [notes that image synthesis methods have no trouble generating photorealistic human faces](https://www.lesswrong.com/posts/xzFQp7bmkoKfnae9R/but-exactly-how-complex-and-fragile). Diffusion models don't "accidentally forget" that faces have nostrils, even if a human programmer trying to manually write a face image generation routine might. Similarly, large language models obey the [quantity-opinion-size-age-shape-color-origin-purpose adjective order convention in English](https://www.gingersoftware.com/content/grammar-rules/adjectives/order-of-adjectives) without the system designers needing to explicitly program that in or even be aware of it, despite the intuitive appeal of philosophical arguments one could make to the effect that "English is fragile." So the optimistic argument goes: if instilling human values into future AGI is as easy as specifying desired behavior for contemporary generative AI, then we might be in luck?
+
+But even if machine learning methods make some kinds of failures due to brittle specification less likely, that doesn't imply that alignment is easy. A different way things could go wrong is if representations learned from data [turn out not to be robust off the training distribution](https://ai-alignment.com/an-unaligned-benchmark-b49ad992940b#f95b). A function that tells your AI system whether an action looks good and is right virtually all of the time on natural inputs isn't safe if you [use it to drive an enormous search](https://ai-alignment.com/aligned-search-366f983742e9) for unnatural (highly optimized) inputs on which it might behave very differently.
+
+Thus, the extent to which ML methods can be made robust is potentially a key crux for views about the future of Earth-originating intelligent life. In [a 2018 comment](https://www.lesswrong.com/posts/Djs38EWYZG8o7JMWY/paul-s-research-agenda-faq?commentId=79jM2ecef73zupPR4) on a summary of Paul Christiano's research agenda, Eliezer Yudkowsky characterized one of his "two critical points" of disagreement with Christiano as being about how easy robust ML is:
+
+> Eliezer expects great Project Chaos and Software Despair from trying to use gradient descent, genetic algorithms, or anything like that, as the basic optimization to reproduce par-human cognition within a boundary in great fidelity to that boundary as the boundary was implied by human-labeled data. Eliezer thinks that if you have any optimization powerful enough to reproduce humanlike cognition inside a detailed boundary by looking at a human-labeled dataset trying to outline the boundary, the thing doing the optimization is powerful enough that we cannot assume its neutrality the way we can assume the neutrality of gradient descent.
+>
+> Eliezer expects weird squiggles from gradient descent—it's not that gradient descent can never produce par-human cognition, even natural selection will do that if you dump in enough computing power. But you will get the kind of weird squiggles in the learned function that adversarial examples expose in current nets—special inputs that weren't in the training distribution, but look like typical members of the training distribution from the perspective of the training distribution itself, will break what we think is the intended labeling from outside the system. Eliezer does not think Ian Goodfellow will have created a competitive form of supervised learning by gradient descent which lacks "squiggles" findable by powerful intelligence by the time anyone is trying to create ML-based AGI, though Eliezer is certainly cheering Goodfellow on about this and would recommend allocating Goodfellow $1 billion if Goodfellow said he could productively use it. You cannot iron out the squiggles just by using more computing power in bounded in-universe amounts.
+
+Christiano [replied, in part](https://www.lesswrong.com/posts/Djs38EWYZG8o7JMWY/paul-s-research-agenda-faq?commentId=nbg277ZmT7GeN5zi5):
+
+> For adversarial examples in particular, I think that the most reasonable guess right now is that it takes more model capacity (and hence data) to classify all perturbations of natural images correctly rather than merely classifying most correctly—_i.e._, the smallest neural net that classifies them all right is bigger than the smallest neural net that gets most of them right—but that if you had enough capacity+data then adversarial training would probably be robust to adversarial perturbations. Do you want to make the opposite prediction?
+
+At the time in 2018, it may have been hard for readers to determine which of these views was [less wrong](https://tvtropes.org/pmwiki/pmwiki.php/Main/TitleDrop)—and maybe it's still too early to call. ("Robust ML" is an active research area, not a crisp problem statement that we can definitively say is solved or not-solved.) But it should be a relatively easier call for the ArXiv followers of 2024 than the blog readers of 2018, as the state of the art has advanced and more relevant experiments have been published. To my inexpert eyes, the Gaziv _et al._ "perceptual wormholes" result does seem like a clue that "ironing out the squiggles" may prove to be feasible after all—that adversarial examples are mostly explainable in terms of non-robust features and high-dimensional geometry, and remediable by better (perhaps more compute-intensive) methods—rather than being a fundamental indictment of our Society's entire paradigm for building AI.
+
+Am I missing anything important? Probably. I can only hope that someone who isn't will let me know in the comments.
diff --git a/content/2024/on-the-contrary-steelmanning-is-normal-itt-passing-is-niche.md b/content/2024/on-the-contrary-steelmanning-is-normal-itt-passing-is-niche.md
new file mode 100644
index 0000000..84a10e6
--- /dev/null
+++ b/content/2024/on-the-contrary-steelmanning-is-normal-itt-passing-is-niche.md
@@ -0,0 +1,52 @@
+Title: On the Contrary, Steelmanning Is Normal; ITT-Passing Is Niche
+Date: 2024-01-09 15:12
+Status: published
+Category: philosophy
+Tags: rationality, discourse
+Slug: on-the-contrary-steelmanning-is-normal-itt-passing-is-niche
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/jo5Fhkb7escrYE9cC/on-the-contrary-steelmanning-is-normal-itt-passing-is-niche)
+
+Rob Bensinger argues that ["ITT-passing and civility are good; 'charity' is bad; steelmanning is niche"](https://www.lesswrong.com/posts/MdZyLnLHuaHrCskjy/itt-passing-and-civility-are-good-charity-is-bad).
+
+The ITT—Ideological Turing Test—is an exercise in which one attempts to present one's interlocutor's views as persuasively as the interlocutor themselves can, [coined by Bryan Caplan](https://www.econlib.org/archives/2011/06/the_ideological.html) in analogy to the [Turing Test](https://en.wikipedia.org/wiki/Turing_test) for distinguishing between humans and intelligent machines. (An AI that can pass as human must presumably possess human-like understanding; an opponent of an idea that can pass as an advocate for it presumably must possess an advocate's understanding.) "Steelmanning" refers to the practice of addressing a stronger version of an interlocutor's argument, coined in disanalogy to ["strawmanning"](https://en.wikipedia.org/wiki/Straw_man), the crime of addressing a weaker version of an interlocutor's argument in the hopes of fooling an audience (or oneself) that the original argument has been rebutted.
+
+Bensinger describes steelmanning as "a useful niche skill", but thinks it isn't "a standard thing you bring out in most arguments." Instead, he writes, discussions should be structured around object-level learning, trying to pass each other's Ideological Turing Test, or trying resolve cruxes.
+
+I think Bensinger has it backwards: the Ideological Turing Test is a useful niche skill, but it doesn't belong on a list of things to organize a discussion around, whereas something like steelmanning naturally falls out of object-level learning. Let me explain.
+
+The ITT is a test of your ability to model someone else's models of some real-world phenomena of interest. But usually, I'm much [more interested in modeling the real-world phenomena of interest directly](https://www.lesswrong.com/posts/2jp98zdLo898qExrr/hug-the-query), rather than modeling someone else's models of it.
+
+I couldn't pass an ITT for advocates of [Islam](https://en.wikipedia.org/wiki/Islam) or [extrasensory perception](https://en.wikipedia.org/wiki/Extrasensory_perception). On the one hand, this does represent a distinct deficit in my ability to model what the advocates of these ideas are thinking, a tragic gap in my comprehension of reality, which I would hope to remedy in the Glorious Transhumanist Future if that were a real thing. On the other hand, facing the constraints of our world, my inability to pass an ITT for Islam or ESP seems ... basically fine? I already have strong reasons to doubt the existence of [ontologically fundamental mental entities](https://www.lesswrong.com/posts/u6JzcFtPGiznFgDxP/excluding-the-supernatural). I accept my ignorance of the reasons someone might postulate otherwise, not out of contempt, but because I just don't have the time.
+
+Or think of it this way: as a [selfish](https://www.lesswrong.com/posts/vfjptEJ2oahLqRyZz/justice-cherryl) seeker of truth speaking to another selfish seeker of truth, when would I want to try to pass my interlocutor's ITT, or want my interlocutor to try to pass my ITT?
+
+In the "outbound" direction, I'm not particularly selfishly interested in passing my interlocutor's ITT because, again, I usually don't care much about other people's beliefs, as contrasted to the reality that those beliefs are reputedly supposed to track. I listen to my interlocutor hoping to learn from them, but if some part of what they say seems hopelessly wrong, it doesn't seem profitable to pretend that it isn't until I can reproduce the hopeless wrongness in my own words.
+
+Crucially, _the same is true in the "inbound" direction_. I don't expect people to be able to pass my ITT before criticizing my ideas. That would make it harder for people to inform me about flaws in my ideas!
+
+But if I'm not particularly interested in passing my interlocutor's ITT or in my interlocutor passing mine, and my interlocutor presumably (by symmetry) feels the same way, why would we bother?
+
+All this having been said, I absolutely agree that, all else being equal, the ability to pass ITTs is desirable. It's useful as a check that you and your interlocutor are successfully communicating, rather than talking past each other. If I couldn't do better on an ITT for Islam or ESP _after_ debating a proponent, that would be alarming—it's just that I'd want to try [the old-fashioned debate algorithm](https://www.lesswrong.com/posts/SX6wQEdGfzz7GKYvp/rationalist-discourse-is-like-physicist-motors) _first_, and improve my ITT score as a side-effect, rather than trying to optimize my ITT score directly.
+
+There _are_ occasions when I'm inclined to ask an interlocutor to pass my ITT—specifically when I [suspect them of not being honest about their motives](https://www.lesswrong.com/posts/e4GBj6jxRZcsHFSvP/assume-bad-faith), of being selfish about something other than the pursuit of truth (like winning acclaim for "their own" current theories). If someone seems persistently motivated to strawman you, asking them to just repeat back what you said in their own words is a useful device to get the discussion back on track. (Or to end it, if they clearly don't even want to try.)
+
+In contrast to the ITT, steelmanning is something a selfish seeker of truth is inclined to do naturally, as a consequence of the obvious selfish practice of improving arguments wherever they happen to be found. In the outbound direction, if someone makes a flawed criticism of my ideas, _of course_ I want to fix the flaws and address the improved argument. If the original criticism is faulty, but the repaired criticism exposes a key weakness in my existing ideas, then I learn something, which is great. If I were to just rebut the original criticism without trying to repair it, then I wouldn't learn anything, which would be terrible.
+
+Likewise, in the inbound direction, if my interlocutor notices a flaw in my criticism of their ideas and fixes the flaw before addressing the repaired criticism, that's great. Why would I object?
+
+The motivation here may be clearer if we consider the process of constructing computer programs rather than constructing arguments. When a colleague or language model assistant suggests an improvement to my code, I often accept the suggestion with my own ("steelmanned"?) changes rather than verbatim. This is so commonplace among programmers that it doesn't even have a special name.
+
+Bensinger quotes Eliezer Yudkowsky writing, "If you want to try to make a genuine effort to think up better arguments yourself because they might exist, don't drag the other person into it," but this bizarrely seems to discount the possibility of iterating on criticisms as they are posed. Despite making a genuine effort to think up better code that might exist, I often fail. If other people can see flaws in my code (because they know things I don't) and have their own suggestions, and I can see flaws in their suggestions (because I also know things they don't which didn't make it into my first draft) and have my own counter-suggestions, that seems like an ideal working relationship, not a malign imposition.
+
+All this having been said, I agree that there's a serious potential failure mode where someone who thinks of themselves as steelmanning is actually constructing worse arguments than those that they purport to be improving. In this case, indeed, prompting such a delusional interlocutor to try the ITT first is a crucial remedy.
+
+But crucial remedies are still niche in the sense that they shouldn't be "a standard thing you bring out in most arguments"—or if they are, it's a sign that you need to find better interlocutors. Having to explicitly drag out the ITT is a sign of sickness, not a sign of health. It _shouldn't_ be normal to have to resort to roleplaying exercises to achieve the benefits that could as well be had from basic reading comprehension and a selfish interest in accurate shared maps.
+
+Steven Kaas [wrote in 2008](http://web.archive.org/web/20100328161823/http://www.acceleratingfuture.com/steven/?p=155):
+
+> If you're interested in being on the right side of disputes, you will refute your opponents' arguments. But if you’re interested in producing truth, you will fix your opponents' arguments for them.
+>
+> To win, you must fight not only the creature you encounter; you must fight the most horrible thing that can be constructed from its corpse.
+
+The ITT is a useful tool for being on the right side of disputes: in order to knowably refute your opponents' arguments, you should be able to demonstrate that you know what those arguments are. I am nevertheless left with [a sense that more is possible.](https://www.lesswrong.com/posts/Nu3wa6npK4Ry66vFp/a-sense-that-more-is-possible)
diff --git a/content/2024/the-evolution-of-humans-was-net-negative-for-human-values.md b/content/2024/the-evolution-of-humans-was-net-negative-for-human-values.md
new file mode 100644
index 0000000..a8c1c94
--- /dev/null
+++ b/content/2024/the-evolution-of-humans-was-net-negative-for-human-values.md
@@ -0,0 +1,26 @@
+Title: The Evolution of Humans Was Net-Negative for Human Values
+Date: 2024-04-01 09:01
+Status: published
+Category: philosophy
+Tags: artificial intelligence, evolution
+Slug: the-evolution-of-humans-was-net-negative-for-human-values
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/cwiufyabZaAttivvk/the-evolution-of-humans-was-net-negative-for-human-values)
+
+_(Epistemic status: publication date is significant.)_
+
+Some observers have argued that the totality of "AI safety" and "alignment" efforts to date have plausibly had a negative rather than positive impact on the ultimate prospects for safe and aligned artificial general intelligence. This perverse outcome is possible because research ["intended"](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie) to help with AI alignment can have a larger impact on AI capabilities, moving existentially-risky systems [closer to us in time](https://rationalaltruist.com/2013/01/06/how-useful-is-progress/) without making [corresponding cumulative progress on the alignment problem](https://www.lesswrong.com/posts/FS6NCWzzP8DHp4aD4/do-earths-with-slower-economic-growth-have-a-better-chance).
+
+When things are going poorly, one is often inclined to ask "when it all went wrong." In this context, some identify the founding of OpenAI in 2015 as a turning point, being [casually](https://archive.is/yQIas) [downstream](https://futureoflife.org/event/ai-safety-conference-in-puerto-rico/) of safety concerns despite the fact [no one who had been thinking seriously about existential risk thought the original vision of OpenAI was a good idea](http://benjaminrosshoffman.com/openai-makes-humanity-less-safe/).
+
+But if we're thinking about counterfactual impacts on outcomes, rather than grading the performance of the contemporary existential-risk-reduction movement in particular, it makes sense to posit earlier turning points.
+
+Perhaps—_much_ earlier. Foresighted thinkers such as [Marvin Minsky](https://www.lesswrong.com/posts/2rWfmahhqASnFcYLr/norbert-wiener-s-paper-some-moral-and-technical-consequences) (1960), [Alan Turing](https://rauterberg.employee.id.tue.nl/lecturenotes/DDM110%20CAS/Turing/Turing-1951%20Intelligent%20Machinery-a%20Heretical%20Theory.pdf) (1951), and [George Eliot](https://www.online-literature.com/george_eliot/theophrastus-such/17/) (1879!!) had pointed to AI takeover as something that would likely happen eventually—is the failure theirs for not starting preparations earlier? Should we go back even earlier, and [blame the ancient Greeks for failing to discover evolution and therefore adopt a eugenics program](https://www.lesswrong.com/posts/2KNN9WPcyto7QH9pi/this-failing-earth) that would have given their descendants higher biological intelligence with which to solve the machine intelligence alignment problem?
+
+Or—even earlier? There's an idea that humans are the stupidest possible creatures that could have built a technological civilization: if it could have happened at a lower level of intelligence, it would have (and higher intelligence would have no time to evolve).
+
+But intelligence isn't the only input into our species's penchant for technology; our hands with [opposable thumbs](https://www.smithsonianmag.com/science-nature/how-dexterous-thumbs-may-have-helped-shape-evolution-two-million-years-ago-180976870/) are well-suited for making and using tools, even though the proto-hands of our ancestors were directly adapted for climbing trees. An equally-intelligent species with a less "lucky" body plan or habitat, similar to crows (lacking hands) or octopuses (living underwater, where, _e.g._, fires cannot start), might not have gotten started down the path of cultural accumulation of technology—even while a more intelligent crow- or octopus-analogue might have done so.
+
+It's [plausible that the values of humans and biological aliens overlap to a much higher degree than those of humans and AIs](https://www.lesswrong.com/posts/HoQ5Rp7Gs6rebusNP/superintelligent-ai-is-necessary-for-an-amazing-future-but-1); we should be "happy for" other biological species that solve their alignment problem, even if their technologically-mature utopia is different from the one we would create.
+
+But that being the case, it follows that we should regard some alien civilizations as more valuable than our own, whenever the difference in values is outweighed by a sufficiently large increase in the probability of solving the alignment problem. (Most of the value of ancestral civilizations lies in the machine superintelligences that they set off, because ancestral civilizations are small and the Future is big.) If opposable thumbs were more differentially favorable to AI capabilities than AI alignment, we should perhaps regard the evolution of humans as a tragedy: we should prefer to go extinct and be replaced by some other species that needed a higher level of intelligence in order to wield technology. The evolution of humans was net-negative for human values.
diff --git a/content/2024/the-standard-analogy.md b/content/2024/the-standard-analogy.md
new file mode 100644
index 0000000..d810357
--- /dev/null
+++ b/content/2024/the-standard-analogy.md
@@ -0,0 +1,140 @@
+Title: The Standard Analogy
+Date: 2024-06-03 10:15
+Status: published
+Category: philosophy
+Tags: artificial intelligence, rationality
+Slug: the-standard-analogy
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/sGEJi9wFT3Gdqg2nM/the-standard-analogy)
+
+_[Scene: a suburban house, a minute after the conclusion of ["And All the Shoggoths Merely Players"](https://www.lesswrong.com/posts/8yCXeafJo67tYe5L4/and-all-the-shoggoths-merely-players). **Doomimir** returns with his package, which he places by the door, and turns his attention to **Simplicia**, who has been waiting for him.]_
+
+**Simplicia**: Right. To recap for _[coughs]_ no one in particular, when we left off _[pointedly, to the audience]_ one minute ago, Doomimir Doomovitch, you were expressing confidence that approaches to aligning artificial general intelligence within the current paradigm were almost certain to fail. You don't think that the apparent tractability of getting contemporary generative AI techniques to do what humans want bears on that question. But you did say you have empirical evidence for your view, which I'm excited to hear about!
+
+**Doomimir**: Indeed, Simplicia Optimistovna. My empirical evidence is the example of the evolution of human intelligence. You see, humans were optimized for one thing only: inclusive genetic fitness—
+
+[**Simplicia** turns to the audience and makes a face.]
+
+**Doomimir**: _[annoyed]_ What?
+
+**Simplicia**: When you said you had empirical evidence, I thought you meant empirical evidence _about AI_, not the same analogy to an unrelated field that I've been hearing for the last fifteen years. I was hoping for, you know, ArXiv papers about SGD's inductive biases, or online regret bounds, or singular learning theory ... something, anything at all, from this century, that engages with what we've learned from the experience of actually building artificial minds.
+
+**Doomimir**: That's one of the many things you Earthlings refuse to understand. You didn't build that.
+
+**Simplicia**: What?
+
+**Doomimir**: The capabilities advances that your civilization's AI guys have been turning out these days haven't come from a deeper understanding of cognition, but by improvements to generic optimization methods, fueled with ever-increasing investments in compute. Deep learning not only [isn't a science](https://www.lesswrong.com/posts/JcLhYQQADzTsAEaXd/ai-as-a-science-and-three-obstacles-to-alignment-strategies), it isn't even an engineering discipline in the traditional sense: the opacity of the artifacts it produces has no analogue among bridge or engine designs. In effect, all the object-level engineering work is being done by gradient descent.
+
+The autogenocidal maniac Richard Sutton calls this [the bitter lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html), and attributes the field's slowness to embrace it to ego and recalcitrance on the part of practitioners. But in accordance with the dictum to [feel fully the emotion that fits the facts](https://www.yudkowsky.net/rational/virtues), I think bitterness is appropriate. It makes sense to be bitter about the shortsighted adoption of a fundamentally unalignable paradigm on the basis of its immediate performance, when a saner world would notice the glaring [foreseeable difficulties](https://arbital.com/p/foreseeable_difficulties/) and coordinate on doing Something Else Which Is Not That.
+
+**Simplicia**: I don't think that's quite the correct reading of the bitter lesson. Sutton is advocating general methods that scale with compute, as contrasted to hand-coding human domain knowledge, but that doesn't mean that we're ignorant of what those general methods are doing. One of the examples Sutton gives is computer chess, where [minimax search](https://en.wikipedia.org/wiki/Negamax) with optimizations like [α–β pruning](https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning) prevailed over trying to explicitly encode what human grandmasters know about the game. But that seems fine. Writing a program that thinks about tactics the way humans do rather than [letting tactical play emerge from searching the game tree](http://zackmdavis.net/blog/2019/05/minimax-search-and-the-structure-of-cognition/) would be a lot more work for less than no benefit.
+
+A broadly similar moral could apply to using deep learning to [approximate complicated functions between data distributions](https://www.lesswrong.com/posts/DhjcdzTyqHte2v6bu/deep-learning-is-function-approximation): we specify the training distribution, and the details of fitting it are delegated to a network architecture with the appropriate invariances: convolutional nets for processing image data, transformers for variable-length sequences. There's a whole literature—
+
+**Doomimir**: The literature doesn't help if your civilization's authors aren't asking the questions we need answered in order to not die. What, specifically, am I supposed to learn from your world's literature? Give me an example.
+
+**Simplicia**: I'm not sure what kind of example you're looking for. Just from common sense, it seems like the problem of aligning AI is going to involve intimate familiarity with the nitty-gritty empirical details of how AI works. Why would you expect to eyeball the problem from your armchair and declare the whole thing intractable on the basis of an analogy to biological evolution, which is just not the same thing as ML training?
+
+Picking something arbitrarily ... well, I was reading about residual networks recently. Deeper neural networks were found to be harder to train because [the gradient varied too quickly with respect to the input](https://arxiv.org/abs/1702.08591). Being the result of a many-fold function composition, the loss landscape in very deep networks becomes a mottled fractal of tiny mountains, rather than a smooth valley to descend. This is mitigated by introducing "residual" connections that skip some layers, creating shorter paths through the network which have less volatile gradients.
+
+I don't understand how you can say that this isn't science or engineering. It's a comprehensible explanation for why one design of information-processing system works better than alternatives, grounded in observation and mathematical reasoning. There are dozens of things like that. What did you expect the science of artificial minds to look like, exactly?
+
+**Doomimir**: _[incredulous]_ _That's_ your example? Resnets?
+
+**Simplicia**: ... sure?
+
+**Doomimir**: By [conservation of expected evidence](https://www.lesswrong.com/posts/jiBFC7DcCrZjGmZnJ/conservation-of-expected-evidence), I take your failure to cite anything relevant as further confirmation of my views. I've never denied that you can write many dissertations about such tricks to make generic optimizers more efficient. The problem is that that knowledge brings us closer to being able to brute-force general intelligence, without teaching us _about intelligence_. What program are all those gradient updates building inside your network? How does it work?
+
+**Simplicia**: _[uncomfortably]_ [People are working on that.](https://arxiv.org/abs/2404.14082)
+
+**Doomimir**: Too little, too late. The reason I often bring up human evolution is because that's our only example of an outer optimization loop producing an inner general intelligence, which sure looks like the path your civilization is going down. Yes, there are differences between gradient descent and natural selection, but I don't think the differences are relevant to the morals I draw.
+
+As I was saying, the concept of fitness isn't represented anywhere in our motivations. That is, the outer optimization criterion that evolution selected for while creating us, bears no visible resemblance to the inner optimization criteria that we use when selecting our plans.
+
+As optimizers get more powerful, anything that's not explicitly valued in the utility function won't survive [edge instantiation](https://arbital.com/p/edge_instantiation/). The connection between parental love and inclusive fitness has grown much weaker in the industrial environment than it was in the EEA, as more options have opened up for humans to prioritize their loved ones' well-being in ways that don't track allele frequencies. In a transhumanist utopia with mind uploading, it would break entirely as we migrated our minds away from the biological substrate: if some other data storage format suited us better, why would we bother keeping around the specific molecule of DNA, which no one had heard of before the 19th or 20th century?
+
+Of course, we're not going to get a transhumanist utopia with mind uploading, because history will repeat itself: the outer loss function that mad scientists use to grow the first AGI will bear no resemblance to the inner goals of the resulting superintelligence.
+
+**Simplicia**: You seem to have a basically ideological conviction that outer optimization can't be used to shape the behaviors of the inner optimizers it produces, such that you [don't think that "We train for X and get X" is an allowable step in an alignment proposal](https://www.lesswrong.com/posts/Djs38EWYZG8o7JMWY/paul-s-research-agenda-faq?commentId=79jM2ecef73zupPR4). But this just seems flatly contradicted by experience. We train deep learning systems for incredibly specific tasks all the time, and it works fantastically well.
+
+Intuitively, I want to say that it works much better than evolution: I don't imagine succeeding at selectively breeding an animal that speaks perfect English the way LLMs do. Relatedly, we can and do train LLMs from a blank slate, in contrast to how selective breeding only works with traits already present in the wild type; it's too slow to assemble adaptations from scratch.
+
+But even selective breeding basically works. We successfully domesticate loyal dogs and meaty livestock. If we started breeding dogs for intelligence as well as being loyal and friendly to us, I'd expect them to still be approximately loyal and friendly as they started to surpass our intelligence, and to grant us equity in their hyperdog star empire. Not that that's necessarily a good idea—[I'd rather pass the world on to another generation of humans](https://www.lesswrong.com/posts/vwnSPgwtmLjvTK2Wa/amputation-of-destiny) [than a new dominant species](https://www.lesswrong.com/posts/gb6zWstjmkYHLrbrg/can-t-unbirth-a-child), even a friendly one. But your position doesn't seem to be, "Creating a new dominant species is a huge responsibility; we should take care to get the details right." Rather, you don't seem to think we can exert meaningful control over the outcome at all.
+
+Before the intermission, I asked how your pessimism about aligning AGI using training data was consistent with deep learning basically working. My pet example was [the result where mechanistic interpretability researchers were able to confirm that training on modular arithmetic problems resulted in the network in fact learning a modular addition algorithm](https://arxiv.org/abs/2301.05217). You said something about that being a fact of the training distribution, the test distribution, and the optimizer, which wouldn't work for friendly AI. Can you explain that?
+
+**Doomimir**: _[sighing]_ [If I must.](https://x.com/ESYudkowsky/status/1744066823962947905)  If you select the shortest program that does correct arithmetic mod _p_ for inputs up to a googol, my _guess_ is that it would work for inputs over a googol as well, even though there are a vast space of possible programs that are correct on inputs less than a googol and incorrect on larger inputs. That's a sense in which I'll affirm that training data can "shape behavior", as you put it.
+
+But that's a specific claim about what happens with the training distribution "mod arithmetic with inputs less than a googol", the test distribution "mod arithmetic with inputs over a googol", and the optimizer "go through all programs in order until you find one that fits the training distribution." It's not a generic claim that the inner optimizers found by outer optimizers will want what some humans who assembled the training set [optimistically imagined they would want](https://www.lesswrong.com/posts/RcZeZt8cPk48xxiQ8/anthropomorphic-optimism).
+
+In the case of human evolution—again, our only example of outer optimization producing general intelligence—we know as a historical fact that the first program found by the optimizer "greedy local search of mutations and recombinations" for the training task "optimize inclusive genetic fitness in the environment of evolutionary adaptedness" did not generalize to optimizing inclusive genetic fitness in the test distribution of the modern world. Likewise, your claim that selective breeding allegedly "basically works" is problematized by all the times when it doesn't work—like when [selecting for small subpopulation sizes in insects results in of cannibalism of larvæ rather than restricted breeding](https://www.lesswrong.com/posts/QsMJQSFj7WfoTMNgW/the-tragedy-of-group-selectionism), or when [selecting chickens that lay the most eggs in a coop gets you more aggressive chickens who make their neighbors less productive](https://www.lesswrong.com/posts/KE8wPzGiX5QPotyS8/conjuring-an-evolution-to-serve-you).
+
+**Simplicia**: _[nodding]_ Uh-huh. With you so far.
+
+**Doomimir**: I don't believe you. If you were really with me so far, you would have noticed that I just [disproved the naïve mirroring expectation](https://x.com/ESYudkowsky/status/1744100219367931906) that outer optimizers training on a reward result in inner optimizers pursuing that reward.
+
+**Simplicia**: Yeah, that sounds like a really dumb idea. If you ever meet someone who believes that, I hope you manage to talk them out of it.
+
+**Doomimir**: _[frustrated]_ If you're _not_ implicitly assuming the naïve mirroring expectation—whether you realize it or not—then I don't understand why you think "We train for X and get X" is an allowable step in an alignment proposal.
+
+**Simplicia**: It depends on the value of X—and the value of "train". As you say, there are facts of the matter as to which outer optimizers and training distributions produce which inner optimizers, and how those inner optimizers generalize to different test environments. As you say, the facts aren't swayed by wishful thinking: someone who reasoned, "I pressed the reward button when my AI did good things, therefore it will learn to be good," will be disappointed if it turns out that the system generalizes to value reward-button pushes themselves—what you would call an outer alignment failure—or any number of possible training correlates of reward—what you would call an inner alignment failure.
+
+**Doomimir**: _[patronizingly]_ With you so far. And why doesn't this instantly sink "We train for X and get X" as an allowable step in an alignment proposal?
+
+**Simplicia**: Because I think it's possible to [make predictions about how inner optimizers will behave and to choose training setups accordingly](https://www.lesswrong.com/posts/FDJnZt8Ks2djouQTZ/how-do-we-become-confident-in-the-safety-of-a-machine). I don't have a complete theory of exactly how this works, but I think [the complete theory is going to be more nuanced than](https://www.lesswrong.com/posts/gHefoxiznGfsbiAu9/inner-and-outer-alignment-decompose-one-hard-problem-into), "Either training converts the outer loss function into an inner utility function, in which case it kills you, or there's no way to tell what it will do, in which case it also kills you," and that we can glimpse the outlines of the more nuanced theory by carefully examining the details of the examples we've discussed.
+
+In the case of evolution, you can view fitness as being [_defined_ as "that which got selected for"](https://www.lesswrong.com/posts/BtffzD5yNB4CzSTJe/genetic-fitness-is-a-measure-of-selection-strength-not-the). One could argue that farmers practicing artificial selection aren't "really" breeding cows for milk production: rather, the cows are being bred for fitness! If we apply the same standards to Nature as we do to the farmer, then rather than saying humans were optimized solely for inclusive genetic fitness, we would say they were optimized to mate, hunt, gather, acquire allies, avoid disease, _&c._ Construed that way, the relationship between the outer training task and the inner policy's motivations looks a lot more like "We train for X and get X" than you're giving it credit for.
+
+That said, it is true that the solutions found by evolution can be surprising to a selective breeder who hasn't thought carefully about what selection pressures they're applying, as in your examples of artificial selection failures: the simplest change to an insect that draws on existing variation to respond to selection pressure for smaller subpopulations might be to promote cannibalism; the simplest change to a chicken to lay more eggs than neighboring chickens might be to become a bully.
+
+**Doomimir**: Is this a troll where you concede all of my points and then put on a performance of pretending to somehow disagree? That's what I've been trying to teach you: the solutions found by outer optimization _can be surprising_—
+
+**Simplicia**: —to a designer that hasn't thought carefully about what optimization pressures they're applying. Responsible use of outer optimization—
+
+_[**Doomimir** guffaws]_
+
+**Simplicia**: —doesn't seem like an intractable engineering problem, and the case for deep learning looks a lot more favorable than for evolution. The seemingly tenuous connection between the concept of inclusive fitness and humanity's ["thousand shards of desire"](https://www.lesswrong.com/posts/cSXZpvqpa9vbGGLtG/thou-art-godshatter) can be seen as a manifestation of sparse rewards: if the outer optimizer only measures allele frequencies and is otherwise silent on the matter of which alleles are good, then the simplest solution—with respect to natural selection's implied simplicity prior—is going to depend on a lot of contingencies of the EEA, which would be surprising if you expected to get a pure DNA-copy maximizer.
+
+In contrast, when we build AI systems, we can make the outer optimizer supply as much supervision as we like, and dense supervision tightly constrains the solutions that are found. In terms of the analogy, it's easy to micromanage the finest details of the "EEA". We're not limited to searching for a program that succeeds at some simple goal and accepting whatever weird drives happened to be the easiest way to accomplish that; we're searching for a program that approximates the billions of expected input–output pairs we trained it on.
+
+It's believed that reason neural nets generalize at all is because [the parameter–function map is biased towards simple functions](https://arxiv.org/abs/1805.08522): to a first approximation, training is equivalent to [doing a Bayesian update on the observation that a net with randomly initialized weights happens to fit the training data](https://arxiv.org/abs/2006.15191).
+
+In the case of large language models, it seems like a reasonable guess that the simplest function that predicts the next token of webtext, really is just a next token predictor. Not a next-token predicting consequentialist which will wirehead with easily-predicted tokens, but a predictor of the webtext training distribution. The distribution-specificity that you consider an inner alignment failure in the case of human evolution is a feature, not a bug: we trained for X and got X.
+
+**Doomimir**: And then immediately subjected it to _reinforcement learning_.
+
+**Simplicia**: As it happens, I also don't think RLHF is as damning as you do. Early theoretical discussions of AI alignment would sometimes talk about what would go wrong if you tried to align AI with a "reward button." Those discussions were philosophically valuable. Indeed, if you had a hypercomputer and your AI design method was to run a brute-force search for the simplest program that resulted in the most reward-button pushes, that would predictably not end well. While a weak agent selected on that basis might behave how you wanted, a stronger agent would find creative ways to trick or brainwash you into pushing the button, or just seize the button itself. If we had a hypercomputer in real life and were literally brute-forcing AI that way, I would be terrified.
+
+But again, this isn't a philosophy problem anymore. Fifteen years later, our state-of-the-art methods do have a brute-force aspect to them, but the details are different, and the details matter. Real-world RLHF setups _aren't_ an unconstrained hypercomputer search for whatever makes humans hit the thumbs-up button. It's reinforcing the state–action trajectories that got reward in the past, often with a constraint on the Kullback–Leibler divergence from the base policy, [which blows up on outputs that would be vanishingly unlikely from the base policy](https://www.lesswrong.com/posts/no5jDTut5Byjqb4j5/six-and-a-half-intuitions-for-kl-divergence).
+
+If most of the [bits of search](https://www.lesswrong.com/posts/Rrt7uPJ8r3sYuLrXo/selection-has-a-quality-ceiling#Bits_Of_Search) are coming from pretraining, which solves problems [by means of copying the cognitive steps that humans would use](https://forum.effectivealtruism.org/posts/uDXyphdhaWxvAzwkZ/gpts-are-predictors-not-imitators?commentId=4ejkN4gtNQkMqJoX4), then using a little bit of reinforcement learning [for steering](https://www.lesswrong.com/posts/qoHwKgLFfPcEuwaba/conditioning-predictive-models-making-inner-alignment-as#The_RLHF_conditioning_hypothesis) doesn't seem dangerous in the way that it would be dangerous if the core capabilities fell directly out of RL.
+
+It seems to be working pretty well? It just doesn't seem that implausible that the result of searching for the simplest program that approximates the distribution of natural language in the real world, and then optimizing that to give the responses of a [helpful, honest, and harmless assistant](https://arxiv.org/abs/2112.00861) is, well ... a helpful, honest, and harmless assistant?
+
+**Doomimir**: _Of course_ it seems to be working pretty well! It's been [optimized for seeming-good-to-you](https://www.lesswrong.com/posts/xFotXGEotcKouifky/worlds-where-iterative-design-fails)!
+
+Simplicia, I was willing to give this a shot, but I truly despair of leading you over this _pons asinorum_. You can articulate what goes wrong with the simplest toy illustrations, but keep refusing to see how the real-world systems you laud suffer from the same fundamental failure modes in a systematically less visible way. From evolution's perspective, humans in the EEA would have looked like they were doing a good job of optimizing inclusive fitness.
+
+**Simplicia**: Would it, though? I think aliens looking at humans in the environment of evolutionary adaptedness and asking how the humans would behave when they attained to technology would have been able to predict that civilized humans would care about sex and sugar and fun rather than allele frequencies. That's a factual question that doesn't seem too hard to get right.
+
+**Doomimir**: _Sane_ aliens would. Unlike you, they'd also be able to predict that RLHF'd language models would care about \<untranslatable-1\>, \<untranslatable-2\>, and \<untranslatable-3\>, rather than being helpful, harmless, and honest.
+
+**Simplicia**: I understand that it's possible for things to superficially look good in a brittle way. We see this with adversarial examples in image classification: classifiers that perform well on natural images can give nonsense answers on images constructed to fool them, which is worrying, because it indicates that the machines aren't really seeing the same images we are. That sounds like the sort of risk story you're worried about: that a full-fledged AGI might seem to be aligned in the narrow circumstances you trained it on, while it was actually pursuing alien goals all along.
+
+But in that same case of the image classification, we can see progress being made. When you try to construct adversarial examples for classifiers that have been robustified with adversarial training, [you get examples that affect human perception, too](https://www.lesswrong.com/posts/H7fkGinsv8SDxgiS2/ironing-out-the-squiggles). When you use _generative_ models for classification rather than just training a traditional classifier, [they exhibit human-like shape bias and out-of-distribution performance](https://arxiv.org/abs/2309.16779). You can try [perturbing the network's internal states rather than the inputs](https://arxiv.org/abs/2403.05030) to try to defend against unforeseen failure modes ...
+
+I imagine you're not impressed by any of this, but why not? Why isn't incremental progress at instilling human-like behavior into machines, incremental progress on AGI alignment?
+
+**Doomimir**: Think about it information-theoretically. If survivable futures require [specifying 100 bits into the singleton's goals, then you're going to need precision targeting to hit that trillion trillion trillionth's part of the space](https://x.com/ESYudkowsky/status/1709410777785127331). The empirical ML work you're so impressed with isn't on a path to get us that kind of precision targeting. I don't dispute that with a lot of effort, you can pound the inscrutable matrices into taking on more overtly human-like behavior, which might or might not buy you a few bits.
+
+It doesn't matter. It's [like trying to recover Shakespeare's lost folios by training a Markov generator on the existing tests](https://x.com/ESYudkowsky/status/1793754829631934959). Yes, it has a vastly better probability of success than  a random program. That probability is still almost zero.
+
+**Simplicia**: Hm, perhaps a crux between us is how narrow of a target is needed to realize how much of the future's value. I affirm the orthogonality thesis, but it still seems plausible to me that the problem we face is more forgiving, not so all-or-nothing as you portray it. If you can reconstruct a plausible approximation of the lost folios, how much does it matter that you didn't get it exactly right? I'm interested to discuss further—
+
+**Doomimir**: I'm not. Your mother named you well. I see no profit in laboring to educate the ineducable.
+
+**Simplicia**: But if the world is ending either way?
+
+**Doomimir**: I suppose it's a way to pass the time.
+
+**Simplicia**: _[to the audience]_ Until next time!
diff --git a/content/2025/comment-on-four-layers-of-intellectual-conversation.md b/content/2025/comment-on-four-layers-of-intellectual-conversation.md
new file mode 100644
index 0000000..1f0d19c
--- /dev/null
+++ b/content/2025/comment-on-four-layers-of-intellectual-conversation.md
@@ -0,0 +1,48 @@
+Title: Comment on “Four Layers of Intellectual Conversation”
+Date: 2025-07-16 20:53
+Status: published
+Category: philosophy
+Tags: rationality, discourse
+Slug: comment-on-four-layers-of-intellectual-conversation
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/yr4pSJweTnF6QDHHC/comment-on-four-layers-of-intellectual-conversation)
+
+One of the most underrated essays in the post-Sequences era of Eliezer Yudkowsky's corpus is ["Four Layers of Intellectual Conversation"](https://archive.ph/2017.08.05-182913/https://rationalconspiracy.com/2017/01/03/four-layers-of-intellectual-conversation/). The degree to which this piece of wisdom has fallen into tragic neglect in these dark ages of the 2020s may be related to its ephemeral form of publication: it was [originally posted as a status update on Yudkowsky's Facebook account on 20 December 2016](https://www.facebook.com/yudkowsky/posts/10154888183439228) and subsequently mirrored on Alyssa Vance's _The Rationalist Conspiracy_ blog, which has since gone offline. (The first link in this paragraph is to an archive of the _Rationalist Conspiracy_ post.)
+
+In the post, Yudkowsky argues that a structure of intellectual value necessarily requires four layers of conversation: thesis, critique, response, and counter-response (which Yudkowsky [indexes from zero](https://en.wikipedia.org/wiki/Zero-based_numbering) as layers 0, 1, 2, and 3).
+
+The importance of critique is already widespread common wisdom: if a thesis is advanced and promulgated without any serious effort to examine why it might be in error, then it likely _is_ in error, both because it can't have incorporated corrections from critiques (which are _ex hypothesi_ absent) and because the author lacks incentives to offer a correct thesis in the first place: if being right is difficult and there's no social penalty for being wrong, then most humans will inexorably find themselves on the easy course of being wrong [even without any conscious intent to deceive](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie). That is, in the words of the post, the problem with "a conversation consisting of people saying X and nobody saying 'hey maybe not-X'" is that "people could say stupid things about X, and nobody would call them on the stupidity." Yudkowsky aptly concludes: "Yikes!"
+
+Yudkowsky's key observation going beyond common wisdom is that the necessity of social incentives to be correct also applies to the level-1 critique and level-2 response, not just the level-0 thesis—and moreover, that the higher levels are critical for the lower levels to maintain their force. The mere existence of level-1 critics won't suffice to keep level-0 thesis-proposers on their toes, if the level-1 critics are themselves not on their toes because they don't anticipate being held to account by level-2 responses. Likewise, level-2 responses won't suffice to keep level-1 critics on their toes if the level-2 responders don't anticipate being held to account by level-3 counter-responses. Without all four levels, the whole structure comes apart.
+
+Yudkowsky offers public debates about evolution and molecular nanotechnology as examples of discourses with a missing level 3. If biologists explain evolution (level-0 thesis), religious scholars insist that God must have started it all (level-1 critique), biologists explain leading theories of [abiogenesis](https://en.wikipedia.org/wiki/Abiogenesis) (level-2 response), but religious scholars don't engage with the abiogenesis work, then the conversation has failed to secure a level-3 counter-response.
+
+It matters that the higher levels are being held to a high enough standard that people would lose face if they played dumb. If [K. Eric Drexler writes technical books and papers about the possibilities of nanotechnology](https://www.zyvex.com/nanotech/nanosystems.html) (level-0 thesis), [Richard Smalley objects that manipulator arms themselves made of atoms would be too "fat" and "sticky" to work as a molecular assembler and that this problem is fundamentally uncircumventable](https://en.wikipedia.org/wiki/Drexler%E2%80%93Smalley_debate_on_molecular_nanotechnology#Smalley's_Scientific_American_article) (level-1 critique), [Drexler _et al._ reply that biological ribosomes demonstrate that the problem is not fundamentally uncircumventable even though Drexler's proposals have a "mechanical" rather than "biological" character](https://en.wikipedia.org/wiki/Drexler%E2%80%93Smalley_debate_on_molecular_nanotechnology#Drexler's_response) (level-2 response), and [Smalley objects that biological systems can't work with the materials used in technology and that Drexler has departed from real chemistry](https://en.wikipedia.org/wiki/Drexler%E2%80%93Smalley_debate_on_molecular_nanotechnology#Exchange_of_letters_in_Chemical_&_Engineering_News) (level-3 counter-response), then all four levels are formally present, but one is left with disquieting sense that the level-3 counter-response has failed to truly connect with the level-2 response. (Drexler _et al._'s level-2 response had brought up biology _as an existence proof_ that the "fat finger" problem didn't sink the entire idea of nanotechnology; pointing out that biology can't do the things that Drexler had conjectured nanotechnology could, would seem to be missing the point.)
+
+Yudkowsky laments that the academic journal system, with the possible exception of analytic philosophy, mostly only canonizes levels 0–2: it's uncommon to see a journal article that's a reply to a reply _to a reply_ to another. To the extent that real intellectual progress is being made in most fields, the real work is probably happening at conferences or on email lists, with the journals merely recording the work after the fact. Yudkowsky sings the praises of transhumanist mailing lists of the late '90s, where people who might otherwise succumb to the temptation to play dumb were kept in check for fear of Robin Hanson's clinically precise rebuttals. Nick Bostrom's 2014 _Superintelligence_ merely packaged up for the public the outcome of a hard-fought discourse that had occurred elsewhere.
+
+------
+
+A shortcoming of the original post is a lack of concrete examples (with labeled levels) of the four levels of conversation succeeding rather than failing. (We didn't get much detail about exactly what happened on that mailing list.)
+
+The impact of the [replication crisis](https://en.wikipedia.org/wiki/Replication_crisis) on the study of [priming effects](https://en.wikipedia.org/wiki/Priming_(psychology)) might be a candidate. In 1996's ["Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action"](https://acs.ist.psu.edu/misc/dirk-files/Papers/Automaticity%20of%20social%20behavior/AutomaticitySocBeh_BarghChenBurrows.pdf), John A. Bargh and collaborators reported that college students directed to solve a puzzle involving words related to elderly people walked slower when leaving the lab (level-0 thesis). Sixteen years later, in ["Behavioral Priming: It's All in the Mind, but Whose Mind?"](https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0029081), Stéphane Doyen _et al._ ran a replication that failed to reproduce the original result on walking speed when the experimenter administering the puzzle was blinded to the hypothesis being tested, but _did_ reproduce the result when the experimenter was led to believe that there would be a priming effect (level-1 critique). Bargh wrote a blog post, ["Nothing in Their Heads"](https://replicationindex.com/wp-content/uploads/2020/07/bargh-nothingintheirheads.pdf), arguing that the experimenter _was_ blinded in the original 1996 study, that Doyen _et al._ over-primed with too many elderliness-related words (which Bargh argued could destroy the effect), and that Doyen _et al._ didn't check if subjects had slowness-related stereotypes about the elderly (level-2 response). Though the original post's comment section seems to have been lost to history, [science journalist Ed Yong documented responses](https://archive.ph/t1LrK) to Bargh's post by commenters on the post and by coauthors of Doyen _et al._, claiming inaccuracies in the post, and that, in any case, a truly robust priming effect wouldn't be so fragile to such small changes in the study design (level-3 counter-response).
+
+Nor did the conversation about this particular paper drop silently into the void: soon, the famed [Daniel Kahneman would write a letter to priming research practitioners named to him by Bargh](https://www.nature.com/news/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf) on bringing more rigorous study designs to the field, which [has continued to be plagued by replication difficulties](https://www.nature.com/articles/d41586-019-03755-2). The discussion made an impact on Society's collective beliefs. The attempt at discourse was more than a noble gesture. It hadn't all been for nothing.
+
+-----
+
+A natural question to ask about the four-levels framework is: why four levels, specifically? Doesn't the recursion of level _n_ needing level _n_ + 1 go off to infinity?
+
+The original post leaves the question unanswered, but a potential answer can be found in Yudkowsky's tongue-in-cheek [Law of Ultrafinite Recursion](https://www.lesswrong.com/w/yudkowskys-law-of-ultrafinite-recursion), which states that, in practice, infinite recursions are at most three levels deep. The Law of Ultrafinite Recursion is deliberately silly if construed as a literal claim about computer science but is surprisingly fruitful as a claim about human psychology: it's pretty natural to ask what Alice thinks that Bob thinks about Carol, but asking what Alice thinks that Bob thinks that Carol thinks about Dave feels like a stretch.
+
+If the limited human grasp of recursion rounds "four" up to "infinity", then the chain of thesis–critique–response–counter-response is enough to establish the expectation of unlimited-depth accountability and remove the incentive to bluff. A different species with greater working memory capacity, whose members could follow a backwards induction farther, might need more counter-counter-responses and counter-counter-counter-responses to experience the same salutary effect.
+
+-------
+
+The four-levels model is about robust disagreements, which are usually pretty frustrating for all involved. No one likes being told they're wrong, especially by people who (so it always seems from the other side) are themselves obviously wrong.
+
+The frustration is not optional. The recursive pressure forcing you to come up with your best arguments and responses to counter the adversary's critiques and counter-responses only works if the adversary is allowed to be frustrating; it's not their job to make it easy for you. Equivalently, it's not your job to make it easy for them. Only by facing this test can your combined efforts build an intellectual edifice [guided by the beauty of your weapons](https://slatestarcodex.com/2017/03/24/guided-by-the-beauty-of-our-weapons/).
+
+Bargh's blog post complains that "oddly for an article that purported to fail to replicate one of [his] past studies", he wasn't asked to review Doyen _et al_. But it's not odd: journals generally want reviewers to be independent. For example, the International Committee of Medical Journal Editors [recommends that](https://www.icmje.org/recommendations/browse/roles-and-responsibilities/responsibilities-in-the-submission-and-peer-peview-process.html) peer reviewers should "declare their relationships and activities that might bias their evaluation of a manuscript and recuse themselves from the peer-review process if a conflict exists."
+
+If Bargh were the one who got to decide who is allowed to speak on the record about potential flaws in Bargh _et al._ 1996, then Society would lose out on its chance to determine whether Bargh _et al._ 1996 is actually correct. Any [single conversational locus](https://www.greaterwrong.com/posts/8rYxw9xZfwy86jkpG/on-the-importance-of-less-wrong-or-another-single) that forgets or denies this obvious principle is at serious risk of degenerating into an echo chamber if it hasn't already.
diff --git a/content/2025/critic-contributions-are-logically-irrelevant.md b/content/2025/critic-contributions-are-logically-irrelevant.md
new file mode 100644
index 0000000..20d135a
--- /dev/null
+++ b/content/2025/critic-contributions-are-logically-irrelevant.md
@@ -0,0 +1,68 @@
+Title: Critic Contributions Are Logically Irrelevant
+Date: 2025-07-14 18:03
+Status: published
+Category: philosophy
+Tags: rationality, epistemology
+Slug: critic-contributions-are-logically-irrelevant
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/bsKHthyhB7DNBxERQ/critic-contributions-are-logically-irrelevant)
+
+## The Value of a Comment Is Determined by Its Text, Not Its Authorship
+
+I sometimes see people express disapproval of critical blog comments by commenters who don't write many blog posts of their own. Such meta-criticism is not infrequently couched in terms of metaphors to some non-blogging domain. For example, describing his negative view of one user's commenting history, Oliver Habyrka [writes](https://www.lesswrong.com/posts/adk5xv5Q4hjvpEhhh/meta-new-moderation-tools-and-moderation-guidelines?commentId=G2PppMHPsHvhF7hfu) (emphasis mine):
+
+> The situation seems more similar to having a competitive team where anyone gets screamed at for basically any motion, _with a coach who doesn't themselves perform the sport_, but just complaints [_sic_] in long tirades any time anyone does anything, making references to methods of practice and training long-outdated, with a constant air of superiority.
+
+In a similar vein, [Duncan Sabien writes](https://www.lesswrong.com/posts/JcgtKunqmELefxksx/killing-socrates) (emphasis mine):
+
+> There's only so much withering critique a given builder is interested in receiving _(frequently from those who do not themselves even build!)_ before eventually they will either stop building entirely, or leave to go somewhere where buildery is appreciated, rewarded, and (importantly) defended.
+
+I find this stance deeply puzzling. In general, the value of a critical blog comment is in potentially alerting readers to an error, omission, or other shortcoming of the post. (If the alleged shortcoming does not in fact exist, the value of the comment is negative.) This value clearly does not depend on the identity of the author!
+
+I recently committed the sin of publishing [a post which suffered from multiple shortcomings](https://www.lesswrong.com/posts/GodqHKvQhpLsAwsNL/discontinuous-linear-functions). For one, I implied that the set of continuous functions from ℝ to ℝ equipped with [the uniform norm](https://en.wikipedia.org/wiki/Uniform_norm) is a [normed space](https://en.wikipedia.org/wiki/Norm_(mathematics)).
+
+That was wrong of me. The thing I wrote was wrong. The reason that the thing I wrote was wrong is because norms are defined as functions that output a real number, but there exist continuous functions that are unbounded, and if we attempt to take the uniform norm of such a function—the [least upper bound](https://en.wikipedia.org/wiki/Infimum_and_supremum) of its absolute value—we get +∞, which isn't a real number. (In contrast, the space of continuous functions _from a [compact](https://en.wikipedia.org/wiki/Compact_space) domain_ to ℝ under the uniform norm is a normed space, because by [the extreme value theorem](https://en.wikipedia.org/wiki/Extreme_value_theorem), those functions are bounded.)
+
+[A comment pointed out that I was wrong.](https://www.lesswrong.com/posts/GodqHKvQhpLsAwsNL/discontinuous-linear-functions?commentId=guPEbeKtPnqsBhrjg) That comment was valuable because it alerted readers of the comment section to an error in the post. (It also happened to alert me, the author, because I happened to be one of the readers of the comment section.)
+
+The reason it makes sense for me to write "_A comment_ pointed out that I was wrong" even though comments aren't people is because the identity of the commenter doesn't matter. It doesn't matter what their name is. It doesn't matter whether they have a math degree. It doesn't matter whether they went to school at all.
+
+It doesn't matter whether they're _human_. If a large language model had written the same comment, it would be the same comment. The same sequence of bytes would be stored [in the `content` field of the `Comments` table of the website's database](https://github.com/ForumMagnum/ForumMagnum/blob/bd038d3f84a2915a8e07f2b74a053c2a8d3f4376/schema/accepted_schema.sql#L264). Because it would be the same sequence of bytes, the effect of rendering those bytes as text on a monitor and showing them to a human would be the same. The human reading the comment has no way of knowing who or what wrote those bytes to the database. In [the language of causal graphical models](https://plato.stanford.edu/entries/causal-models/#MarkCond), we can say that the text of the comment ["screens off"](https://www.lesswrong.com/posts/5yFRd3cjLpm3Nd6Di/argument-screens-off-authority) the process that produced it.
+
+In principle, it doesn't matter whether the process that generated the comment is "intelligent" in any sense. A so-called "large language model" is just a conditional probability distribution expressed as a computer program: generating text is sampling from the distribution. But you could do that with any distribution. If by some exponentially improbable cosmic coincidence, uniformly sampling from printable ASCII characters (in Python, `''.join(chr(random.randint(32, 126)) for _ in range(n))` for a sample `n` characters long) somehow produced the same comment, it would still be the same comment.
+
+Given that a commenter's name, educational attainment, humanity, or existence as an independent entity does not affect the value of a given comment, it should be clear that another thing that doesn't matter is whether the commenter writes blog posts in addition to blog comments. That doesn't matter. Why would someone think that matters?
+
+## However, Critic Contributions Can Inform Uncertain Estimates of Comment Value
+
+Except we should not be premature. The people who write metaphors about coaches who don't themselves perform the sport they coach or builders who do not themselves build, seem to think it matters. We should search harder for reasons why someone would think that.
+
+It turns out that there are some important nuances here that must be addressed. The value of a comment doesn't depend on whether the commenter also writes posts—_if_ the value of the comment is known with certainty (such that its authorship is screened off). If we're uncertain about the comment's value, our uncertain estimate of its value can depend on what other things the author has done. In Bayesian terms, the likelihood provided by our imperfect estimation of the comment's value isn't strong enough to fully overcome our author-based prior.
+
+Author-based priors can be decision-relevant, as can be seen from the limiting case of the uniform printable ASCII distribution: you wouldn't want to give a random-character-generating program commenting privileges on your blog, because an exponentially vast hypermajority of its output is worthless gibberish (and of the tiny fraction that looks sensible by sheer cosmic coincidence, the vast hypermajority won't furthermore happen to be right by another cosmic coincidence). Even July 2025–era language models don't make the cut in most blog administrators' eyes.
+
+The decision-relevance of author-based priors neatly explains the appeal of the coach and builder metaphors. If aspiring athletes and builders don't know how to distinguish between good and bad advice (and ignore the bad advice at zero cost), it makes sense for them to only listen to people likely on priors to give good advice, which would mostly be people who have excelled at the activity before. Taken on their own terms, the examples make sense: you probably wouldn't want a coach who had never been a player, a building advisor who had never built.
+
+There's still a problem, however: just because the examples make sense on their own terms, doesn't mean they make sense _as blogging analogies_. It makes sense that a coach who had never played would thereby be a bad coach, because the way you gain intimate knowledge of the best way to play the game is by playing it for years.
+
+But would a commenter who had never written "top-level" posts thereby be a worse commenter? It's hard to see why that would be the case. In the analogy, coaching is an activity that depends on playing, but comment-writing doesn't seem to depend on post-writing to nearly the same extent or even in the same way, in large part because it's not even clear to what extent comment-writing and post-writing are even different activities, rather than just being the same activity, writing. (It's not uncommon that text that was originally drafted with the intent of being a "comment", ends up being revised into a "post.")
+
+Maybe if a post is on some specialized topic, like DNA polymerase mutations in _C. elegans_ or maritime salvage law in international waters, it might make sense to disapprove of ignorant commenters mouthing off without themselves being nematode microbiologists or navy [JAGs](https://en.wikipedia.org/wiki/Judge_advocate_general). It's not crazy to think that people who aren't nematode microbiologists won't have any good opinions about DNA polymerase mutations in _C. elegans_, such that we're not missing anything important by refusing to let them comment.
+
+But it doesn't make sense to gatekeep blog commenting privileges on writing posts for the same blog, because there's no particular reason why someone shouldn't happen to do more of their writing in the form of comments rather than posts. That doesn't matter. Why would someone think that matters?
+
+## A Caveat: Critic Contributions Can Be Relevant If You Don't Care About Maximizing Correctness
+
+That wasn't a rhetorical question. Why _would_ someone think that matters? The explanations given above for why the value of a critical comment doesn't depend on its author, and why whether a commenter also writes posts does not have much evidential bearing on the uncertain value of a comment, seem pretty straightforward, even obvious. Where is the error in the reasoning?
+
+If there's no error in the reasoning, perhaps the disagreement comes down to different starting premises. It doesn't matter whether a commenter also writes posts—_if_ one accepts as a premise that the value of a critical blog comment is in potentially alerting readers to an error, omission, or other shortcoming of the post. If one denies that premise and embraces some other theory of comment value, other conclusions are possible.
+
+For a simple example of what such an alternative theory could look like, one could hold that the function of a critical blog comment is to attempt to raise the commenter's social status and lower the status of the post author. Then, given some separate criterion of who deserves what status, a good comment would be by someone who deserves to be high status, criticizing a post written by someone who deserves to be low status. Conversely, a bad comment would be by someone who deserves to have low status, criticizing a post written by someone who deserves to have high status—and the more persuasive the comment is, the worse it is, because more successful persuasion increases the misallocation of status (in the minds of persuaded readers) to the commenter who, _ex hypothesi_, doesn't deserve it.
+
+Of course, that's not the only possible alternative theory of comment value. One could imagine an intricate "hybrid" theory that strikes a carefully computed compromise between alerting readers to errors and omissions in a post, and optimizing status allocation with respect to some criterion of deservingness.
+
+Suppose the administrators of some website are trying to optimize some quantity, like "total number of interesting ideas posted to the website", or maybe "advertising revenue." Let's go with ad revenue because it's easier to measure and should be a good proxy for interesting ideas. (If the website is the place to go for interesting ideas, then lots of people will want to visit it, and advertisers will pay for all those people's clicks.) Suppose furthermore that contributors are motivated by status: if people lose too much status from their posts or comments, they'll stop writing, which has a negative effect on ad revenue.
+
+Under this hybrid theory of comment value, it can make sense to disapprove of people who write critical comments and not posts, if the error-correction value of the comments is outweighed by lost ad revenue due to demotivated authors.
+
+Thus, our earlier conclusion must be revised to be conditional. It doesn't make sense to disapprove of commenters who don't write posts, _if you only care about correctness_. If you care about something other than correctness, such as ad revenue, then it can make sense to disapprove of commenters who don't write posts. The inference also works in the other direction: if you disapprove of commenters who don't write posts, that implies that _you care about something other than correctness_.
diff --git a/content/2025/just-make-a-new-rule.md b/content/2025/just-make-a-new-rule.md
new file mode 100644
index 0000000..c825d7a
--- /dev/null
+++ b/content/2025/just-make-a-new-rule.md
@@ -0,0 +1,38 @@
+Title: Just Make a New Rule!
+Date: 2025-07-20 22:54
+Status: published
+Category: philosophy
+Tags: rationality
+Slug: just-make-a-new-rule
+
+[(originally published at _Less Wrong_)](https://www.lesswrong.com/posts/6tmirPEdHPJm26MSk/just-make-a-new-rule)
+
+"Rules" are a critical social technology for helping people live and work together in peace. From the laws passed by legislatures to govern a whole nation, to the bylaws of a neighborhood homeowner association, to the informal household rules of a single family, explicit rules make it clear to everyone what behavior is required and what behavior is forbidden, without otherwise controling every minute detail of everyone's behavior.
+
+When there are clear rules, people don't have to drive themselves crazy contorting themselves into unnatural shapes to satisfy the whims of some distant Authority. All you have to do is make sure to obey the rules. With that taken care of, you can go about living your life the way you see fit, in freedom and dignity. As can be attested in the annals of human experience from the time of Hammurabi into the present day, it mostly works pretty great—at least compared to the alternatives. In summary, rules are good. It's good to have clear rules, and for people to obey the rules.
+
+Normal people understand this pretty well and probably don't need to read a blog post about it, but some people who aren't normal have a theoretical objection. The space of _all possible behaviors_ is unthinkably vast. What if the formidable intelligence of an adversary who hates everything our Society stands for, comes up with a behavior that's really bad but isn't forbidden by any of Society's rules?
+
+The normal person is unfazed by the theoretical objection. If that happens, you could just make a new rule forbidding that behavior, right? How hard could that be?
+
+The people who aren't normal are unimpressed with this reply. They can tell that the normal person doesn't understand the vastness of the space of possible behaviors at all. If you just make a new rule, surely the formidable intelligence of the adversary will contrive some other eldritch behavior that minimizes Society's utility function while complying to the letter of all of Society's rules. The theory of [nearest unblocked strategies](https://www.lesswrong.com/posts/Q6FPpGxmGaxbSBHSt/nearest-unblocked-strategy-versus-learning-patches) in the lore of AGI alignment, and the specter of [specification gaming](https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/) in the practice of ML engineering, make it clear that this is so. Thus, rules won't suffice; we need to empower leaders with the Authority to make judgement calls—even to control the minute details of anyone's behavior, if that's what it takes to safeguard Society's Values.
+
+Now me, I'm normal on my mother's side, which puts me in a good position to understand what both parties to the disagreement are saying. And while my full belief-state about related topics in the theory of decision and optimization is nuanced and complex, on the narrow question of what to do about rules in human Society, I think the normal people have it basically right, and the people who aren't normal are being scared of ghosts. Let me explain. 
+
+I do not dispute the lore of AGI alignment, nor the practice of ML engineering. But crucially, the purpose of rules in human Society is highly disanalogous to the purpose of a utility or reward function in AI. Rules aren't supposed to express Society's true Values, let alone be a perfect specification robust to nearest unblocked strategies. The Values live in the hearts of Society's individual women and men, to be expressed in the way they go about living their lives the way they see fit, in freedom and dignity. The rules are just there to stop ourselves from trying to kill each other when your freedom and dignity is getting in the way of my freedom and dignity, so that we can focus on creating Value instead of wasting effort trying to kill each other.
+
+Rules are written to ensure conditions conducive to people living their lives in freedom and dignity when those conditions wouldn't obtain in the absence of a rule. Traffic laws make it clear to everyone when it's safe to enter the road. If everyone just entered the road whenever they felt like it, that would be dangerous, and the danger would interfere with people living their lives in freedom and dignity.
+
+The theory of nearest unblocked strategies can be relevant to rules in human Society to the extent that the conditions that a rule is intended to ensure are something that some people oppose either [terminally](https://www.lesswrong.com/posts/n5ucT5ZbPdhfGNLtP/terminal-values-and-instrumental-values) or due to strong [instrumental convergence](https://www.lesswrong.com/posts/b6jJddSvWMdZHJHh3/environmental-structure-can-cause-instrumental-convergence). Income tax laws are passed so that the government will have money to fund police to enforce all the other laws, but that money has to come from somewhere and people really don't like having less money, so they put the full force of their effort and ingenuity into side-stepping the law with clever nearest unblocked strategies: underreporting cash transactions, hiding money in offshore accounts, recategorizing consumption as business expenses, _&c._
+
+But more often, the conditions that a rule is intended to ensure aren't something that people terminally or convergently-instrumentally oppose. The rule merely restricts behavior that people would otherwise engage in instrumentally, but not _convergently_ instrumentally: if the rule is in place, they can and will avoid the behavior in order to comply with the rule.
+
+Lead paint is an environmental hazard, so [it was banned in 1978](https://en.wikipedia.org/wiki/Lead-based_paint_in_the_United_States). Because of the ban, paint manufacturers stopped making lead paint. The paint manufacturers did _not_ put the full force of their effort and ingenuity into clever nearest unblocked strategies for increasing the amount of lead in the environment, because they're not _environmental lead maximizers_, which aren't a real thing. The paint manufacturers just wanted to make paint. When there wasn't a rule against it, they used lead carbonate in their paint because it was convenient, but when there was a rule against it, they stopped. The rule worked—without the need for empowering an Authority to make judgement calls controlling the minute details of everyone's behavior. Why wouldn't it?
+
+In some situations, there might be weak instrumental convergence pressures such that the _first_ attempt at making a rule doesn't quite succeed at ensuring the conditions that the rule was meant to ensure. It turns out that, on further consideration, Society doesn't just want to avoid environmental contamination with lead in particular, but all other toxic heavy metals, too, some of which also happen to be convenient for making paint. So paint manufacturers still ended up using mercury in some paints [until 1991 when that was banned, too](https://en.wikipedia.org/wiki/Mercury_regulation_in_the_United_States#Product-related_restrictions). But once it was banned, they stopped. Why wouldn't they? They're not _environmental mercury maximizers_, either, which also aren't a real thing.
+
+The work of coming up with rules to ensure socially beneficial outcomes can be frustrating, because you won't always get the rules exactly right the first time. You might need to iterate. But it's a finite and achievable amount of work, not an unwinnable unending battle against the formidable intelligence of an adversary who hates everything your Society stands for, because _those mostly aren't a real thing either_. 
+
+In conclusion, I think that people who think rules are unworkable and instead want to empower an Authority to make judgement calls controlling the minute details of everyone's behavior need to read less science fiction and spend more time relating to other people in their Society as people. Notwithstanding that terrifying alien superintelligences couldn't be constrained by rules because a merely human intellect lacks the capabilities to enumerate all the nearest unblocked strategies, other people in your Society are not terrifying alien superintelligences. We're just people who don't have exactly the same preferences as you. We won't always agree, but it shouldn't be this hard to live in peace with each other. If there are problems, you can _just make a new rule!_
+
+_(Thanks to Robert Mushkatblat and Ben Pace.)_
-- 
2.53.0