{"id":2463,"date":"2026-01-27T07:17:20","date_gmt":"2026-01-27T15:17:20","guid":{"rendered":"http:\/\/zackmdavis.net\/blog\/?p=2463"},"modified":"2026-01-27T07:20:19","modified_gmt":"2026-01-27T15:20:19","slug":"disagreement-comes-from-the-dark-world","status":"publish","type":"post","link":"http:\/\/zackmdavis.net\/blog\/2026\/01\/disagreement-comes-from-the-dark-world\/","title":{"rendered":"Disagreement Comes From the Dark World"},"content":{"rendered":"

In \u201cTruth or Dare\u201d<\/a>, Duncan Sabien articulates a phenomenon in which expectations of good or bad behavior can become self-fulfilling: people who expect to be exploited and feel the need to put up defenses both elicit and get sorted into a Dark World where exploitation is likely and defenses are necessary, whereas people who expect beneficence tend to attract beneficence in turn.<\/p>\n

Among many other examples, Sabien highlights the phenomenon of gift economies<\/a>: a high-trust culture in which everyone is eager to help each other out whenever they can is a nicer place to live than a low-trust culture in which every transaction must be carefully tracked for fear of enabling free-riders.<\/p>\n

I\u2019m skeptical of the extent to which differences between high- and low-trust cultures can be explained by self-fulfilling prophecies as opposed to pre-existing differences in trust-worthiness<\/em>, but I do grant that self-fulfilling expectations can sometimes play a role: if I insist on always being paid back immediately and in full, it makes sense that that would impede the development of gift-economy culture among my immediate contacts. So far, the theory articulated in the essay seems broadly plausible.<\/p>\n

Later, however, the post takes an unexpected turn:<\/p>\n

Treating all of the essay thus far as prerequisite and context:<\/p>\n

This is why you should not trust<\/em> Zack Davis, when he tries to tell you what constitutes good conduct<\/a> and productive<\/a> discourse<\/a>. Zack Davis does not understand how high-trust, high-cooperation dynamics work. He has never seen them. They are utterly outside of his experience and beyond his comprehension. What he knows how to do is keep his footing in a world of liars and thieves and pickpockets, and he does this with genuinely admirable skill and inexhaustible tenacity.<\/p>\n

But (as far as I can tell, from many interactions across years) Zack Davis does not understand how advocating for and deploying those survival tactics<\/em> (which are 100% appropriate for use in an adversarial memetic environment) utterly destroys the possibility of building something Better<\/em>. Even if he wanted<\/em> to hit the \u201ccooperate\u201d button\u2014<\/p>\n

(In contrast to his usual stance, which from my perspective is something like \u201clook, if we all hit \u2018defect\u2019 together<\/em>, in full foreknowledge, then we don\u2019t have to extend trust in any direction and there\u2019s no possibility of any unpleasant surprises and you can all stop grumping at me for repeatedly \u2018defecting\u2019 because we\u2019ll all be cooperating on the meta level, it\u2019s not like I didn\u2019t warn<\/em> you which button I was planning on pressing, I am in fact very consistent and conscientious.\u201d)<\/p>\n

\u2014I don\u2019t think he knows where it is, or how to press it.<\/p>\n

(Here I\u2019m talking about the literal actual Zack Davis, but I\u2019m also using him as a stand-in for all the dark world denizens whose well-meaning advice fails to take into account the possibility of light.)<\/p><\/blockquote>\n

As a reader of the essay, I reply: wait, who? Am I supposed to know who this Davies person is? Ctrl-F search confirms that they weren\u2019t mentioned earlier in the piece; there\u2019s no reason for me to have any context for whatever this section is about.<\/p>\n

As Zack Davis, however, I have a more specific reply, which is: yeah, I don\u2019t think that button does what you think it does. Let me explain.<\/p>\n

\n

In figuring out what would constitute good conduct and productive discourse, it\u2019s important to appreciate how bizarre the human practice of \u201cdiscourse\u201d looks in light of Aumann\u2019s dangerous idea<\/a>.<\/p>\n

There\u2019s only one<\/a> reality. If I\u2019m a Bayesian reasoner honestly reporting my beliefs about some question, and you\u2019re also a Bayesian reasoner honestly reporting your beliefs about the same question, we should converge on the same answer, not because we\u2019re cooperating with each other, but because it is<\/em> the answer. When I update my beliefs based on your report on your beliefs, it\u2019s strictly because I expect your report to be evidentially entangled<\/a> with the answer. Maybe that\u2019s a kind of \u201ctrust\u201d, but if so, it\u2019s in the same sense in which I \u201ctrust\u201d that an increase in atmospheric pressure will exert force on the exposed basin of a classical barometer and push more mercury up the reading tube<\/a>. It\u2019s not personal and it\u2019s not reciprocal: the barometer and I aren\u2019t doing each other any favors. What would that even mean?<\/p>\n

In contrast, my friends and I in a gift economy are<\/em> doing each other favors. That kind of setting featuring agents with a mixture of shared and conflicting interests is the context in which the concepts of \u201ccooperation\u201d and \u201cdefection\u201d and reciprocal \u201ctrust\u201d (in the sense of people trusting each other, rather than a Bayesian robot trusting a barometer) make sense. If everyone pitches in with chores when they can, we all get the benefits of the chores being done\u2014that\u2019s cooperation. If you never wash the dishes, you\u2019re getting the benefits of a clean kitchen without paying the costs\u2014that\u2019s defection. If I retaliate by refusing to wash any dishes myself, then we both suffer a dirty kitchen, but at least I\u2019m not being exploited\u2014that\u2019s mutual defection. If we institute a chore wheel with an auditing regime, that re\u00ebstablishes cooperation, but we\u2019re paying higher transaction costs for our lack of trust. And so on: Sabien\u2019s essay does a good job of explaining how there can be more than one possible equilibrium in this kind of system, some of which are much more pleasant than others.<\/p>\n

If you\u2019ve seen high-trust gift-economy-like cultures working well and low-trust backstabby cultures working poorly, it might be tempting to generalize from the domains of interpersonal or economic relationships, to rational (or even \u201crationalist\u201d<\/a>) discourse. If trust and cooperation are essential for living and working together, shouldn\u2019t the same lessons apply straightforwardly to finding out what\u2019s true together?<\/p>\n

Actually, no. The issue is that the payoff matrices are different.<\/p>\n

Life and work involve a mixture of shared and conflicting interests. The existence of some conflicting interests is an essential part of what it means for you and me to be two different agents rather than interchangable parts of the same hivemind: we should hope to do well together, but when push comes to shove, I care more about me doing well than you doing well. The art of cooperation is about maintaining the conditions such that push does not in fact come to shove.<\/p>\n

But correct epistemology does not involve conflicting interests. There\u2019s only one reality. Bayesian reasoners cannot agree to disagree. Accordingly, when humans successfully approach the Bayesian ideal, it doesn\u2019t particularly feel like cooperating with your beloved friends, who see you with all your blemishes and imperfections but would never let a mere disagreement interfere with loving you. It usually feels like just perceiving things\u2014resolving disagreements so quickly that you don\u2019t even notice them as disagreements<\/a>.<\/p>\n

Suppose you and I have just arrived at a bus stop. The bus arrives every half-hour. I don\u2019t know when the last bus was, so I don\u2019t know when the next bus will be: I assign a uniform probability distribution over the next thirty minutes. You recently looked at the transit authority\u2019s published schedule, which says the bus will come in six minutes: most of your probability-mass is concentrated tightly around six minutes from now.<\/p>\n

We might not consciously notice this as a \u201cdisagreement\u201d, but it is: you and I have different beliefs about when the next bus will arrive; our probability distributions aren\u2019t the same. It\u2019s also very ephemeral: when I ask, \u201cWhen do you think the bus will come?\u201d and you say, \u201csix minutes; I just checked the schedule\u201d, I immediately replace my belief with yours, because I think the published schedule is probably right and there\u2019s no particular reason for you to lie about what it says.<\/p>\n

Alternatively, suppose that we both checked different versions of the schedule, which disagree: the schedule I looked at said the next bus is in twenty minutes, not six. When we discover the discrepancy, we infer that one of the schedules must have been outdated, and both adopt a distribution with most of the probability-mass in separate clumps around six and twenty minutes from now. Our initial beliefs can\u2019t both have been right\u2014but there\u2019s no reason for me to weight my prior belief more heavily just because it was mine.<\/p>\n

At worst, approximating ideal belief exchange feels like working on math. Suppose you and I are studying the theory of functions of a complex variable. We\u2019re trying to prove or disprove the proposition that if an entire function satisfies f<\/em>(x<\/em>\u2005+\u20051)\u2004=\u2004f<\/em>(x<\/em>)<\/span> for real x<\/em><\/span>, then f<\/em>(z<\/em>\u2005+\u20051)\u2004=\u2004f<\/em>(z<\/em>)<\/span> for all complex z<\/em><\/span>. I suspect the proposition is false and set about trying to construct a counterexample; you suspect the proposition is true and set about trying to write a proof by contradiction. Our different approaches do seem to imply different probabilistic beliefs about the proposition, but I can\u2019t be confident in my strategy just because it\u2019s mine, and we expect the disagreement to be transient: as soon as I find my counterexample or you find your reductio<\/em>, we should be able to share our work and converge.<\/p>\n

\n

Most real-world disagreements of interest don\u2019t look like the bus arrival or math problem examples\u2014qualitatively, not as a matter of trying to prove quantitatively harder theorems. Real-world disagreements tend to persist; they\u2019re predictable\u2014in flagrant contradiction of how the beliefs of Bayesian reasoners would follow a random walk<\/a>. From this we can infer that typical human disagreements aren\u2019t \u201chonest\u201d<\/a>, in the sense that at least one of the participants is behaving as if they have some other goal than getting to the truth.<\/p>\n

Importantly, this characterization of dishonesty is using a functionalist criterion<\/a>: when I say that people are behaving as if they have some other goal than getting to the truth, that need not imply that anyone is consciously lying<\/a>; \u201cmere\u201d bias is sufficient to carry the argument.<\/p>\n

Dishonest disagreements end up looking like conflicts because they are disguised conflicts<\/a>. The parties to a dishonest disagreement are competing to get their preferred belief accepted, where beliefs are being preferred for some reason other than their accuracy: for example, because acceptance of the belief would imply actions that would benefit the belief-holder. If it were true<\/em> that my company is the best, it would follow logically that customers should buy my products and investors should fund me. And yet a discussion with me about whether or not my company is the best probably doesn\u2019t feel like a discussion about bus arrival times or the theory of functions of a complex variable. You probably expect me to behave as if I thought my belief is better \u201cbecause it\u2019s mine\u201d, to treat attacks on the belief<\/em> as if they were attacks on my person: a conflict rather than a disagreement.<\/p>\n

\u201cMy company is the best\u201d is a particularly stark example of a typically dishonest belief, but the pattern is very general: when people are attached to their beliefs for whatever reason\u2014which is true for most of the beliefs that people spend time disagreeing about, as contrasted to math and bus-schedule disagreements that resolve quickly\u2014neither party is being rational (which doesn\u2019t mean neither party is right on the object level). Attempts to improve the situation should take into account that the typical case is not that of truthseekers who can do better at their shared goal if they learn to trust each other, but rather of people who don\u2019t trust each other because each correctly<\/em> perceives that the other is not<\/em> truthseeking.<\/p>\n

Again, \u201cnot truthseeking\u201d here is meant in a functionalist sense. It doesn\u2019t matter if both parties subjectively think of themselves as honest. The \u201cdistrust\u201d that prevents Aumann-agreement-like convergence is about how agents respond to evidence, not about subjective feelings. It applies as much to a mislabeled barometer as it does to a human with a functionally-dishonest belief. If I don\u2019t think the barometer readings correspond to the true atmospheric pressure, I might still update on evidence from the barometer in some way if I have a guess about how its labels correspond to reality, but I\u2019m still going to disagree with its reading according to the false labels.<\/p>\n

\n

There are techniques for resolving economic or interpersonal conflicts that involve both parties adopting a more cooperative approach, each being more willing to do what the other party wants (while the other reciprocates by doing more of what the first one wants). Someone who had experience resolving interpersonal conflicts using techniques to improve cooperation might be tempted to apply the same toolkit to resolving dishonest disagreements.<\/p>\n

It might very well work for resolving the disagreement. It probably doesn\u2019t work for resolving the disagreement correctly<\/em>, because cooperation is about finding a compromise amongst agents with partially conflicting interests, and in a dishonest disagreement in which both parties have non-epistemic goals, trying to do more of what the other party functionally \u201cwants\u201d amounts to catering to their bias, not systematically getting closer to the truth.<\/p>\n

Cooperative approaches are particularly dangerous insofar as they seem likely to produce a convincing but false illusion of rationality, despite the participants\u2019 best of subjective conscious intentions. It\u2019s common for discussions to involve more than one point of disagreement. An apparently productive discussion might end with me saying, \u201cOkay, I see you have a point about X, but I was still right about Y.\u201d<\/p>\n

This is a success if the reason I\u2019m saying that is downstream of you in fact having a point about X but me in fact having been right about Y. But another state of affairs that would result in me saying that sentence, is that we were functionally playing a social game in which I implicitly agreed to concede on X (which you visibly care about) in exchange for you ceding ground on Y (which I visibly care about).<\/p>\n

Let\u2019s sketch out a toy model to make this more concrete. \u201cTruth or Dare\u201d uses color perception as an illustration of confirmation bias<\/a>: if you\u2019ve been primed to make the color yellow salient, it\u2019s easy to perceive an image as being yellower than it is.<\/p>\n

Suppose Jade and Ruby consciously identify as truthseekers, but really, Jade is biased to perceive non-green things as green 20% of the time, and Ruby is biased to perceive non-red things as red 20% of the time. In our functionalist sense, we can model Jade as \u201cwanting\u201d to misrepresent the world as being greener than it is, and Ruby as \u201cwanting\u201d to misrepresent the world is being redder than it is.<\/p>\n

Confronted with a sequence of gray objects, Jade and Ruby get into a heated argument: Jade thinks 20% of the objects are green and 0% are red, whereas Ruby thinks they\u2019re 0% green and 20% red.<\/p>\n

As tensions flare, someone who didn\u2019t understand the deep disanalogy between human relations and epistemology might propose that Jade and Ruby should strive be more \u201ccooperative\u201d, establish higher \u201ctrust.\u201d<\/p>\n

What does that mean? Honestly, I\u2019m not entirely sure, but I worry that if someone takes high-trust gift-economy-like cultures as their inspiration and model for how to approach intellectual disputes, they\u2019ll end up giving bad advice in practice.<\/p>\n

Cooperative human relationships result in everyone getting more of what they want. If Jade wants to believe that the world is greener than it is and Ruby wants to believe that the world is redder than it is, then na\u00efve attempts at \u201ccooperation\u201d might involve Jade making an effort to see things Ruby\u2019s way at Ruby\u2019s behest, and vice versa<\/em>. But Ruby is only going to insist that Jade make an effort to see it her way when Jade says an item isn\u2019t red. (That\u2019s what Ruby cares about.) Jade is only going to insist that Ruby make an effort to see it her way when Ruby says an item isn\u2019t green. (That\u2019s what Jade cares about.)<\/p>\n

If the two (perversely) succeed at seeing things the other\u2019s way, they would end up converging on believing that the sequence of objects is 20% green and 20% red (rather than the 0% green and 0% red that it actually is). They\u2019d be happier, but they would also be wrong. In order for the pair to get the correct answer, then without loss of generality, when Ruby says an object is red, Jade needs to stand her ground<\/em>: \u201cNo, it\u2019s not red; no, I don\u2019t trust you and won\u2019t see things your way; let\u2019s break out the Pantone swatches.\u201d But that doesn\u2019t seem very \u201ccooperative\u201d or \u201ctrusting\u201d.<\/p>\n

\n

At this point, a proponent of the high-trust, high-cooperation dynamics that Sabien champions is likely to object that the absurd \u201c20% green, 20% red\u201d mutual-sycophancy outcome in this toy model is clearly not what they meant. (As Sabien takes pains to clarify in \u201cBasics of Rationalist Discourse\u201d<\/a>, \u201cIf two people disagree, it\u2019s tempting for them to attempt to converge with each other<\/em>, but in fact the right move is for both of them to try to see more of what\u2019s true<\/em>.\u201d)<\/p>\n

Obviously, the mutual sycophancy outcome is clearly not what proponents of trust and cooperation consciously intend. The problem is that mutual sycophancy seems to be the natural outcome of treating interpersonal conflicts as analogous to epistemic disagreements and trying to resolve them both using cooperative practices, when in fact the decision-theoretic structure of those situations are very different. The text of \u201cTruth or Dare\u201d seems to treat the analogy as a strong one; it wouldn\u2019t make sense to spend so many thousands of words discussing gift economies and the eponymous party game and then draw a conclusion about \u201cwhat constitutes good conduct and productive discourse\u201d, if gift economies and the party game weren\u2019t relevant to what constitutes productive discourse.<\/p>\n

\u201cTruth or Dare\u201d seems to suggest that it\u2019s possible to escape the Dark World by excluding the bad guys. \u201c[F]rom the perspective of someone with light world privilege, [\u2026] it did not occur to me that you might be hanging around someone with ill intent at all,\u201d Sabien imagines a denizen of the light world saying. \u201cCan you, um. Leave? Send them away? Not be spending time in the vicinity of known or suspected malefactors?\u201d<\/p>\n

If we\u2019re talking about holding my associates to a standard of ideal truthseeking (as contrasted to a lower standard of \u201cnot using this truth-or-dare game to blackmail me\u201d), then, no, I think I\u2019m stuck spending time in the vicinity of people who are known or suspected to be biased. I can try to mitigate the problem by choosing less<\/em> biased friends, but when we do disagree, I have no choice<\/a> but to approach that using the same rules of reasoning that I would use with a possibly-mislabeled barometer, which do not have a particularly cooperative character. Telling us that the right move is for both of us to try to see more of what\u2019s true is tautologically correct but non-actionable; I don\u2019t know how to do<\/em> that except by my usual methodology, which Sabien has criticized as characteristic of living in a dark world.<\/p>\n

That is to say: I do not understand how high-trust, high-cooperation dynamics work. I\u2019ve never seen them. They are utterly outside my experience and beyond my comprehension. What I do know is how to keep my footing in a world of people with different goals from me, which I try to do with what skill and tenacity I can manage.<\/p>\n

And if someone should say that I should not be trusted<\/em> when I try to explain what constitutes good conduct and productive discourse \u2026 well, I agree!<\/p>\n

I don\u2019t want people to trust me, because I think trust would result in us getting the wrong answer.<\/p>\n

I want people to read the words I write, think it through for themselves, and let me know in the comments if I got something wrong.<\/p>\n","protected":false},"excerpt":{"rendered":"

In \u201cTruth or Dare\u201d, Duncan Sabien articulates a phenomenon in which expectations of good or bad behavior can become self-fulfilling: people who expect to be exploited and feel the need to put up defenses both elicit and get sorted into … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[10],"tags":[98,27],"_links":{"self":[{"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/posts\/2463"}],"collection":[{"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/comments?post=2463"}],"version-history":[{"count":2,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/posts\/2463\/revisions"}],"predecessor-version":[{"id":2465,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/posts\/2463\/revisions\/2465"}],"wp:attachment":[{"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/media?parent=2463"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/categories?post=2463"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/tags?post=2463"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}