From: Zack M. Davis Date: Tue, 18 Nov 2025 20:21:28 +0000 (-0800) Subject: move "Disagreement Comes From the Dark World" to this repo X-Git-Url: https://zackmdavis.net/blog/source?a=commitdiff_plain;h=edfcdfc831feb6609f2562f69574066ca6ba4918;p=An_Algorithmic_Lucidity.git move "Disagreement Comes From the Dark World" to this repo --- diff --git a/disagreement_comes_from_the_dark_world.md b/disagreement_comes_from_the_dark_world.md new file mode 100644 index 0000000..77bc2ae --- /dev/null +++ b/disagreement_comes_from_the_dark_world.md @@ -0,0 +1,74 @@ +# Disagreement Comes From the Dark World + +In ["Truth or Dare"](https://homosabiens.substack.com/p/truth-or-dare), Duncan Sabien articulates a phenomenon in which expectations of good or bad behavior can become self-fulfilling: people who expect to be exploited and feel the need to put up defenses both elicit and get sorted into a Dark World where exploitation is likely and defenses are necessary, whereas people who expect beneficience tend to attract beneficience in turn. + +Among many other examples, Sabien highlights the phenomenon of [gift economies](https://en.wikipedia.org/wiki/Gift_economy): a high-trust culture in which everyone is eager to help each other out whenever they can is a nicer place to live than a low-trust culture in which every transaction must be carefully tracked for fear of enabling free-riders. + +I'm skeptical of the extent to which differences between high- and low-trust cultures can be explained by self-fulfilling prophecies as opposed to pre-existing differences in trust-_worthiness_, but I do grant that self-fulfilling expectations can sometimes play a role: if I insist on always being paid back immediately and in full, it makes sense that that would impede the development of gift-economy culture among my immediate contacts. So far, the theory articulated in the essay seems broadly plausible. + +Later, however, the post takes an unexpected turn: + +> Treating all of the essay thus far as prerequisite and context: +> +> This is why you _should not trust_ Zack Davis, when he tries to tell you what constitutes [good conduct](https://www.lesswrong.com/posts/iThwqe3yPog56ytyq/aiming-for-convergence-is-like-discouraging-betting) and [productive](https://www.lesswrong.com/posts/5zjucvhSFvp92eYbE/reply-to-duncan-sabien-on-strawmanning) [discourse](https://www.lesswrong.com/posts/SX6wQEdGfzz7GKYvp/rationalist-discourse-is-like-physicist-motors). Zack Davis does not understand how high-trust, high-cooperation dynamics work. He has never seen them. They are utterly outside of his experience and beyond his comprehension. What he knows how to do is keep his footing in a world of liars and thieves and pickpockets, and he does this with genuinely admirable skill and inexhaustible tenacity. +> +> But (as far as I can tell, from many interactions across years) Zack Davis does not understand how _advocating for and deploying those survival tactics_ (which are 100% appropriate for use in an adversarial memetic environment) _utterly destroys the possibility of building something Better_. Even if he _wanted_ to hit the "cooperate" button— +> +> (In contrast to his usual stance, which from my perspective is something like "look, if we all hit 'defect' _together_, in full foreknowledge, then we don't have to extend trust in any direction and there's no possibility of any unpleasant surprises and you can all stop grumping at me for repeatedly 'defecting' because we'll all be cooperating on the meta level, it's not like I didn't _warn_ you which button I was planning on pressing, I am in fact very consistent and conscientious.") +> +> —I don't think he knows where it is, or how to press it. +> +> (Here I'm talking about the literal actual Zack Davis, but I’m also using him as a stand-in for all the dark world denizens whose well-meaning advice fails to take into account the possibility of light.) + +As a reader of the essay, I reply: wait, who? Am I supposed to know who this Davies person is? Ctrl-F search confirms that they weren't mentioned earlier in the piece; there's no reason for me to have any context for whatever this section is about. + +As Zack Davis, however, I have a more specific reply, which is: yeah, I don't think that button does what you think it does. Let me explain. + +------ + +In figuring out what would constitute good conduct and productive discourse, it's important to appreciate how bizarre the human practice of "discourse" looks in light of [Aumann's dangerous idea](https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem). + +[There's only one](https://www.youtube.com/watch?v=cRd6oF-vBQg) reality. If I'm a Bayesian reasoner honestly reporting my beliefs about some question, and you're also a Bayesian reasoner honestly reporting your beliefs about the same question, we should converge on the same answer, not because we're cooperating with each other, but because it _is_ the answer. When I update my beliefs based on your report on your beliefs, it's strictly because I expect your report to be [evidentially entangled](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence) with the answer. Maybe that's a kind of "trust", but if so, it's in the same sense in which I "trust" that [an increase in atmospheric pressure will exert force on the exposed basin of a classical barometer and push more mercury up the reading tube](https://en.wikipedia.org/wiki/Barometer#Mercury_barometers). It's not personal and it's not reciprocal: the barometer and I aren't doing each other any favors. What would that even mean? + +In contrast, my friends and I in a gift economy _are_ doing each other favors. That kind of setting featuring agents with a mixture of shared and conflicting interests is the context in which the concepts of "cooperation" and "defection" and reciprocal "trust" (in the sense of people trusting each other, rather than a Bayesian robot trusting a barometer) make sense. If everyone pitches in with chores when they can, we all get the benefits of the chores being done—that's cooperation. If you never wash the dishes, you're getting the benefits of a clean kitchen without paying the costs—that's defection. If I retaliate by refusing to wash any dishes myself, then we both suffer a dirty kitchen, but at least I'm not being exploited—that's mutual defection. If we institute a chore wheel with an auditing regime, that reëstablishes cooperation, but we're paying higher transaction costs for our lack of trust. And so on: Sabien's essay does a good job of explaining how there can be more than one possible equilibrium in this kind of system, some of which are much more pleasant than others. + +If you've seen high-trust gift-economy-like cultures working well and low-trust backstabby cultures working poorly, it might be tempting to generalize from the domains of interpersonal or economic relationships, to rational (or even ["rationalist"](https://www.lesswrong.com/posts/SX6wQEdGfzz7GKYvp/rationalist-discourse-is-like-physicist-motors)) discourse. If trust and cooperation are essential for living and working together, shouldn't the same lessons apply straightforwardly to finding out what's true together? + +Actually, no. The issue is that the payoff matrices are different. + +Life and work involve a mixture of shared and conflicting interests. The existence of some conflicting interests is an essential part of what it means for you and me to be two different agents rather than interchangable parts of the same hivemind: we should hope to do well together, but when push comes to shove, I care more about me doing well than you doing well. The art of cooperation is about maintaining the conditions such that push does not in fact come to shove. + +But correct epistemology does not involve conflicting interests. There's only one reality. Bayesian reasoners cannot agree to disagree. Accordingly, when humans successfully approach the Bayesian ideal, it doesn't particularly feel like cooperating with your beloved friends, who see you with all your blemishes and imperfections but would never let a mere disagreement interfere with loving you. It usually feels like just perceiving things—[resolving disagreements so quickly that you don't even notice them as disagreements](https://www.lesswrong.com/posts/4S6zunFNFY3f5JYxt/aumann-agreement-is-common). + +Suppose you and I have just arrived at a bus stop. The bus arrives every half-hour. I don't know when the last bus was, so I don't know when the next bus will be: I assign a uniform probability distribution over the next thirty minutes. You recently looked at the transit authority's published schedule, which says the bus will come in six minutes: most of your probability-mass is concentrated tightly around six minutes from now. + +We might not consciously notice this as a "disagreement", but it is: you and I have different beliefs about when the next bus will arrive; our probability distributions aren't the same. It's also very ephemeral: when I ask, "When do you think the bus will come?" and you say, "six minutes; I just checked the schedule", I immediately replace my belief with yours, because I think the published schedule is probably right and there's no particular reason for you to lie about what it says. + +Alternatively, suppose that we both checked different versions of the schedule, which disagree: the schedule I looked at said the next bus is in twenty minutes, not six. When we discover the discrepancy, we infer that one of the schedules must have been outdated, and both adopt a distribution with most of the probability-mass in separate clumps around six and twenty minutes from now. Our initial beliefs can't both have been right—but there's no reason for me to weight my prior belief more heavily just because it was mine. + +At worst, approximating ideal belief exchange feels like working on math. Suppose you and I are studying the theory of functions of a complex variable. We're trying to prove or disprove the proposition that if an entire function satisfies $f(x + 1) = f(x)$ for real $x$, then $f(z + 1) = f(z)$ for all complex $z$. I suspect the proposition is false and set about trying to construct a counterexample; you suspect the proposition is true and set about trying to write a proof by contradiction. Our different approaches do seem to imply different probabilistic beliefs about the proposition, but I can't be confident in my strategy just because it's mine, and we expect the disagreement to be transient: as soon as I find my counterexample or you find your _reductio_, we should be able to share our work and converge. + +----- + +It's worth noticing that most real-world disagreements of interest don't look like the bus arrival or math problem examples—qualitatively, not as a matter of trying to prove quantitatively harder theorems. Real-world disagreements tend to persist; they're [predictable—in flagrant contradiction of how the beliefs of Bayesian reasoners would follow a random walk](https://www.overcomingbias.com/p/we_cant_foreseehtml). From this we can infer that [typical human disagreements aren't "honest"](https://mason.gmu.edu/~rhanson/deceive.pdf), in the sense that at least one of the participants is behaving as if they have some other goal than getting to the truth. + +Importantly, this characterization of dishonesty is using [a functionalist criterion](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie): when I say that people are behaving as if they have some other goal than getting to the truth, that [need not imply that anyone is consciously lying](https://www.lesswrong.com/posts/bSmgPNS6MTJsunTzS/maybe-lying-doesn-t-exist); "mere" bias is sufficient to carry the argument. + +Dishonest disagreements end up looking like conflicts because they are disguised conflicts. The parties to a dishonest disagreement are competing to get their preferred belief onto shared maps. + +[TODO— + * people fight to get their preferred beliefs onto the shared map (and your preferred belief chosen for some reason other than accuracy is better "because it's yours", it deviates frm truth in a way that benefits your goals if believed) + * There are techniques for resolving undisguised conflicts that involve being more cooperative, willing to do more of what the other party wants (while you reciprocate by doing more of what they want) + * Someone who thought the economic-interpersonal and the epistemic were analogous, might look at disguised conflicts and try to apply the same toolkit + * But the analogy is broken because when both parties have non-epistemic goals, "doing more of what the other person wants" results in validating rather than tearing down each other's self-serving delusions + ] + +[TODO— a toy model of mutual validation vs. mutual teardown] + +------ + +And if someone should say that I _should not be trusted_ when I try to explain what constitutes good conduct and productive discourse ... well, I agree! + +I don't want people to trust me. + +I want people to read the words I write, think it through for themselves, and let me know in the comments if I got something wrong.