Importantly, this characterization of dishonesty is using [a functionalist criterion](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie): when I say that people are behaving as if they have some other goal than getting to the truth, that [need not imply that anyone is consciously lying](https://www.lesswrong.com/posts/bSmgPNS6MTJsunTzS/maybe-lying-doesn-t-exist); "mere" bias is sufficient to carry the argument.
-Dishonest disagreements end up looking like conflicts because they are disguised conflicts. The parties to a dishonest disagreement are competing to get their preferred belief accepted, where beliefs are being preferred for some reason other than their accuracy: for example, because acceptance of the belief would imply actions that would benefit the belief-holder.
+Dishonest disagreements end up looking like conflicts [because they are disguised conflicts](https://www.lesswrong.com/posts/e4GBj6jxRZcsHFSvP/assume-bad-faith). The parties to a dishonest disagreement are competing to get their preferred belief accepted, where beliefs are being preferred for some reason other than their accuracy: for example, because acceptance of the belief would imply actions that would benefit the belief-holder. If it were _true_ that my company is the best, it would follow logically that customers should buy my products and investors should fund me. And yet a discussion with me about whether or not my company is the best probably doesn't feel like a discussion about bus arrival times or the theory of functions of a complex variable. You probably expect me to behave as if I thought my belief is better "because it's mine", to treat attacks on _the belief_ as if they were attacks on my person: a conflict rather than a disagreement.
-If it were _true_ that my company is the best, it would follow logically that customers should buy my products and investors should fund me. And yet a discussion with me about whether or not my company is the best probably doesn't feel like a discussion about bus arrival times or the theory of functions of a complex variable. You probably expect me to behave as if I thought my belief is better "because it's mine."
+"My company is the best" is a particularly stark example of a dishonest belief, but the pattern is very general: when people are attached to their beliefs for whatever reason—which is true for most of the beliefs that people spend time disagreeing about, as contrasted to math and bus-schedule disagreements that resolve quickly—neither party is being rational (which doesn't mean neither party is right on the object level). Attempts to improve the epistemic situation should take into account that the typical situation is not that of truth-seekers who can do better at their shared goal if they learn to trust each other, but rather of people who don't trust each other because each _correctly_ perceives that the other is _not_ truth-seeking.
-to treat attacks on the belief
+Again, "not truth-seeking" here is meant in a functionalist sense. It doesn't matter if both parties subjectively think of themselves as honest. The "distrust" that prevents Aumann-agreement-like convergence is about how agents respond to evidence, not about subjective feelings. It applies as much to a mislabeled barometer as it does to a human with a functionally-dishonest belief. If I don't think the barometer readings correspond to the true atmospheric pressure, I might still update on evidence from the barometer in some way if I have a guess about how its labels correspond to reality, but I'm still going to disagree with its reading according to the false labels.
+----
+There are techniques for resolving economic or interpersonal conflicts that involve both parties adopting a more cooperative approach, each being more willing to do what the other party wants (while the other reciprocates by doing more of what the first one wants). Someone who had experience resolving interpersonal conflicts using techniques to improve cooperation might be tempted to apply the same toolkit to resolving dishonest disagreements.
-[TODO—
- * There are techniques for resolving undisguised conflicts that involve being more cooperative, willing to do more of what the other party wants (while you reciprocate by doing more of what they want)
- * Someone who thought the economic-interpersonal and the epistemic were analogous, might look at disguised conflicts and try to apply the same toolkit
- * But the analogy is broken because when both parties have non-epistemic goals, "doing more of what the other person wants" results in validating rather than tearing down each other's self-serving delusions
- ]
+It might very well work for resolving the disagreement. It probably doesn't work for resolving the disagreement _correctly_, because cooperation is about finding a compromise amongst agents with partially conflicting interests, and in a dishonest disagreement in which both parties have non-epistemic goals, trying to do more of what the other party "wants" (again, in a functionalist sense) amounts to catering to their bias, not systematically getting closer to the truth.
-[TODO— a toy model of mutual validation vs. mutual teardown]
+Cooperative approaches are particularly dangerous insofar as they seem likely to produce a convincing but false illusion of rationality, despite the participants' best of subjective conscious "intentions". It's common for discussions to involve more than one point of disagreement. An apprarently productive discussion might end with me saying, "Okay, I see you have a point about X, but I was still right about Y."
+
+This is a success if the reason I'm saying that is downstream of you in fact having a point about X but me in fact having been right about Y. But another state of affairs that would result in me saying that sentence, is that we were functionally playing a social game in which I implicitly agreed to concede on X (which you visibly care about) in exchange for ceding ground on Y (which I visibly care about).
+
+Let's sketch out a toy model to make this more concrete. "Truth or Dare" [provides an illustration of confirmation bias](https://www.lesswrong.com/posts/TQ4AXj3bCMfrNPTLf/truth-or-dare#Scene_III__Patterns__Projections__and_Preconceptions): if you've been primed to make the color yellow salient, it's easy to perceive an image as being yellower than it is.
+
+Suppose Jade and Ruby consciously identify as truthseekers, but really, Jade is biased to perceive non-green things as green 20% of the time, and Ruby is biased to perceive non-red things as red 20% of the time. In our functionalist sense, we can model Jade as "wanting" to misrepresent the world as being greener than it is, and Ruby as "wanting" to misrepresent the world is being redder than it is.
+
+Confronted with a sequence of gray objects, Jade and Ruby get into a heated argument: Jade thinks 20% of the objects are green and 0% are red, whereas Ruby thinks they're 0% green and 20% red.
+
+As tensions flare, someone who didn't understand the deep disanalogy between human relations and epistemology might propose that Jade and Ruby should strive be more "cooperative", establish higher "trust."
+
+What does that mean? Honestly, I'm not entirely sure, but I worry that if someone takes high-trust gift-economy-like cultures as their inspiration and model for how to approach intellectual disputes, they'll end up giving bad advice in practice.
+
+Cooperative human relationships result in everyone getting more of what they want. If Jade wants to believe that the world is greener than it is and Ruby wants to believe that the world is redder than it is, then naïve attempts at "cooperation" might involve Jade making an effort to see things Ruby's way at Ruby's behest, and _vice versa_. But Ruby is only going to insist that Jade make an effort to see it her way when Jade says an item isn't red. (That's what Ruby cares about.) Jade is only going to insist that Ruby make an effort to see it her way when Ruby says an item isn't green. (That's what Jade cares about.)
+
+If the two (perversely) succeed at seeing things the other's way, they would end up converging on believing that the sequence of objects is 20% green and 20% red (rather than the 0% green and 0% red that it actually is). They'd be happier, but they would also be wrong. In order for the pair to get the correct answer, then without loss of generality, when Ruby says an object is red, Jade needs to _stand her ground_: "No, it's not red; no, I don't trust you and won't see things your way; let's break out the Pantone swatches", but that doesn't seem very "cooperative" or "trusting".
------
+At this point, a proponent of the high-trust, high-cooperation dynamics that Sabien champions is likely to object that the absurd "20% green, 20% red" mutual sycophancy outcome in this toy model is clearly not what they meant. (As Sabien [takes pains to clarify in "Basics of Rationalist Discourse"](https://www.lesswrong.com/posts/XPv4sYrKnPzeJASuk/basics-of-rationalist-discourse-1#5__Aim_for_convergence_on_truth__and_behave_as_if_your_interlocutors_are_also_aiming_for_convergence_on_truth_), "If two people disagree, it's tempting for them to attempt to converge _with each other_, but in fact the right move is for both of them to _try to see more of what's true_.")
+
+I agree that the mutual sycophancy outcome is clearly not what proponents of trust and cooperation consciously intend. The problem is that mutual sycophancy seems to be the natural outcome of treating interpersonal conflicts as analogous to epistemic disagreements and trying to resolve them both using cooperative practices, when in fact the decision-theoretic structure of those situations are very different. The text of "Truth or Dare" seems to treat the analogy as a strong one; it wouldn't make sense to spend so many thousands of words discussing gift economies and the eponymous party game it it weren't relevant to what constitutes productive discourse.
+
+"Truth or Dare" seems to suggest that it's possible to escape the Dark World by excluding the bad guys. "[F]rom the perspective of someone with light world privilege, [...] it did not occur to me that you might be hanging around someone with ill intent at all," Sabien imagines a denizen of the light world saying. "Can you, um. Leave? Send them away? Not be spending time in the vicinity of known or suspected malefactors?"
+
+If we're talking about holding my associates to a standard of ideal truthseeking (as contrasted to the standard of "not using this truth-or-dare game to blackmail me"), then, no, I think I'm stuck spending time in the vicinity of people who are known or suspected to be biased. I can try to mitigate the problem somewhat by choosing _less_ biased friends, but when we do disagree, I [have no choice](https://www.lesswrong.com/posts/eY45uCCX7DdwJ4Jha/no-one-can-exempt-you-from-rationality-s-laws) but to approach that using the same rules of reasoning that I could use with a possibly-mislabeled barometer, which do not have a particularly cooperative character. Telling us that the right move is for both of us to try to see more of what's true is tautologically correct but non-actionable; I don't know how to _do_ that except by the same methodology that Sabien has criticized as characteristic of living in a dark world.
+
+That is to say: I do not understand how high-trust, high-cooperation dynamics work. I've never seen them. They are utterly outside my experience and beyond my comprehension. What I do know is how to keep my footing in a world of people with different goals from me, which I try to do with what skill and tenacity I can manage.
+
And if someone should say that I _should not be trusted_ when I try to explain what constitutes good conduct and productive discourse ... well, I agree!
-I don't want people to trust me.
+I don't want people to trust me, because I think trust would result in us getting the wrong answer.
I want people to read the words I write, think it through for themselves, and let me know in the comments if I got something wrong.
projects / An_Algorithmic_Lucidity.git / commitdiff