Unfortunately, there's a difference between the story everyone and her dog tells herself, and the behavior of everyone and her dog as an information-processing system in the physical universe. The story doesn't necessarily need to be true. There's nothing stopping me from telling myself a story about how I seek out true and useful information, while in fact seeking out information that _feels_ good, but which might actually be false or useless.
-Keeping the story true turns out to be a harder problem than it sounds. Everyone and her dog knows that the map is not the territory, but the reason we need a whole slogan about it is because we never actually have unmediated access to the territory. Everything we think we know about the territory is actually just part of our map (the world-simulation our brains construct from sensory data), which makes it easy to get confused about whether your actions are improving the real territory, or just your view of it on your map.
+Keeping the story true turns out to be a harder problem than it sounds. Everyone and her dog knows that the map is not the territory, but the reason we need a whole slogan about it is because we never actually have unmediated access to the territory. Everything we think we know about the territory is actually just part of our map (the world-simulation our brains construct from sensory data), which makes it easy to lose track of whether your actions are improving the real territory, or just your view of it on your map.
For example, I like it when I have good ideas. It makes sense for me to like that. I endorse taking actions that will result in world-states in which I have good ideas.
The problem is that I might not be able to tell the difference between world-states in which I have good ideas, and world-states in which I _think_ my ideas are good, but they're actually bad. Those two different states of the territory would look the same on my map.
-If my brain's learning algorithms reinforce behaviors that lead to me having ideas that I think are good, then in addition to behaviors that make me have better ideas (like reading a book), I might also inadvertently pick up behaviors that prevent me from hearing about it if my ideas are bad (like silencing critics).
+If my brain's learning algorithms reinforce behaviors that lead to me having ideas that I think are good, then in addition to learning behaviors that make me have better ideas (like reading a book), I might also inadvertently pick up behaviors that prevent me from hearing about it if my ideas are bad (like silencing critics).
-This might seem like an easy problem to solve, because the most basic manifestations of the problem are in fact pretty easy to solve. If I were to throw a crying fit and yell, "Critics bad! No one is allowed to criticize my ideas!" every time someone criticized my ideas, the problem with that would be pretty obvious to everyone and her dog, and I would stop getting invited to the salon.
+This might seem like an easy problem to solve, because the most basic manifestations of the problem are in fact pretty easy to solve. If I were to throw a crying fit and yell, "Critics bad! No one is allowed to criticize my ideas!" every time someone criticized my ideas, the problem with that would be pretty obvious to everyone and her dog, and I would stop getting invited to the [salon](https://en.wikipedia.org/wiki/Salon_(gathering)).
+
+But what if there were subtler manifestations of the problem, that _weren't_ obvious to everyone and her dog? Then I might keep getting invited to the salon, and possibly even spread the covertly dysfunctional behavior to other salon members. (If they saw the behavior seeming to work for me, they might imitate it, and their brain's learning algorithms would reinforce it if it seemed to work for them.) What might those look like? Let's try to imagine.
-But what if there were subtler manifestations of the problem, that _weren't_ obvious to everyone and her dog? Then I might keep getting invited to the salon, and possibly even spread the problematic behavior to other salon members. (If they saw the behavior seeming to work for me, they might imitate it, and their brain's learning algorithms would reinforce it if it seemed to work for them.) What might those look like? Let's think about it.
## Filtering Interlocutors
This one is subtle. Goofusia isn't throwing a crying fit every time a member of the salon criticizes her ideas. And indeed, you can't invite the whole world to your salon. You can't _not_ do some sort of filtering. The question is whether salon invitations are being extended or withheld for "good" reasons (that promote the salon processing true and useful information) or "bad" reasons (that promote false or useless information).
-The problem is that being friends with Goofusia and "know[ing] that [she and other salon members] want the truth" is a bad membership criterion, not a good one, because people who aren't friends with Goofusia and don't know that she wants the truth are likely to have different things to say. Even if Goofusia can answer all the critiques her friends can think of, that shouldn't give her confidence that her ideas are solid, if there are likely to be serious critiques that wouldn't be independently reïnvented by the kinds of people who become Goofusia's friends.
+The problem is that being friends with Goofusia and "know[ing] that [she and other salon members] want the truth" is a bad membership criterion, not a good one, because people who aren't friends with Goofusia and don't know that she wants the truth are likely to have different things to say. Even if Goofusia can answer all the critiques her friends can think of, that shouldn't give her confidence that her ideas are solid, if there are likely to exist serious critiques that wouldn't be independently reïnvented by the kinds of people who become Goofusia's friends.
+
+The "nutrient" metaphor is a tell. Goofusia seems to be thinking of criticism as if it were a homogenous ingredient necessary for a healthy epistemic environment, but that it doesn't particularly matter where it comes from. In analogy, it doesn't matter whether you get your allowance of potassium from bananas or potatoes or artificial supplements. If you find bananas and potatoes unpleasant, you can still take supplements and get your potassium that way; if you find Goody Osborne unpleasant, you can just talk to your friends who know you want the truth and get your criticism that way.
-The "nutrient" metaphor is a tell. Goofusia seems to be thinking of criticism as if it were a homogenous ingredient necessary for a healthy epistemic environment, but that it doesn't particularly matter where it comes from. In analogy, it doesn't matter whether you get your allowance of potassium from bananas or potatoes or artificial supplements. If you find bananas and potatoes unpleasant, you can still take supplements and get your potassium that way.
+But unlike chemically uniform nutrients, criticism isn't homogenous: different critics are differently equipped by virtue of their different intellectual backgrounds to notice different flaws in a piece of work. The purpose of criticism is not to virtuously endure being criticized; the purpose is to surface and fix every individual flaw. (If you independently got everything exactly right the first time, then there would be nothing for critics to do; that just seems pretty unlikely if you're talking about anything remotely complicated. It would be hard to believe that such an unlikely-seeming thing had really happened without the best critics getting the opportunity to do their worst.)
-But unlike chemically uniform nutrients, criticism isn't homogenous: different critics are differently equipped by virtue of their different intellectual backgrounds to notice different flaws in a piece of work. The purpose of criticism is not to virtuously endure being criticized; the purpose is to surface and fix every individual flaw.
+"Knowing that (someone) wants the truth" is a particularly poor filter, because people who think that they have strong criticisms of your ideas [are particularly likely to think that you don't want the truth](https://www.lesswrong.com/posts/iThwqe3yPog56ytyq/aiming-for-convergence-is-like-discouraging-betting). (Because, the reasoning would go, if you did want the truth, why would you propose such flawed ideas, instead of independently inventing the obvious-to-them criticism yourself and dropping the idea without telling anyone?) Refusing to talk to people who think that they have strong criticisms of your ideas is a bad thing to do if you care about your ideas being correct.
-"Knowing that (someone) wants the truth" is a particularly poor filter, because people who think that they have strong criticisms of your ideas are particularly likely to think that you don't want the truth. (Because, the reasoning would go, [if you did want the truth, why would you propose such flawed ideas, instead of independently inventing the obvious-to-them criticism yourself](https://www.lesswrong.com/posts/iThwqe3yPog56ytyq/aiming-for-convergence-is-like-discouraging-betting) and dropping the idea without telling anyone?) Refusing to talk to people who think that they have strong criticisms of your ideas is a bad thing to do if you care about your ideas being correct.
+The selection effect is particularly bad in situations where the fact that someone doesn't want the truth is relevant to the correct answer. Suppose Goofusia proposes that the salon buys cookies from a certain bakery—which happens to be owned by Goofusia's niece. If Goofusia's proposal was motivated by nepotism, that's [probabilistically relevant](https://www.lesswrong.com/posts/y4bkJTtG3s5d6v36k/stupidity-and-dishonesty-explain-each-other-away) to evaluating the quality of the proposal. (If the salon members aren't omniscient at evaluating bakery quality on the merits, then they can be deceived by recommendations made for reasons other than the merits.) The salon can debate back and forth about the costs and benefits of spending the salon's snack budget at the niece's bakery, but if no one present is capable of thinking "Maybe Goofusia is being nepotistic" (because anyone who could think that would never be invited to Goofusia's salon), that bodes poorly for the salon's prospects of understanding the true cost–benefit landscape of catering options.
-The effect is particularly bad in situations where the fact that someone doesn't want the truth is relevant to the correct answer. Suppose Goofusia proposes that the salon buys cookies from a certain bakery—which happens to be owned by Goofusia's niece. If Goofusia's proposal was motivated by nepotism, that's [probabilistically relevant](https://www.lesswrong.com/posts/y4bkJTtG3s5d6v36k/stupidity-and-dishonesty-explain-each-other-away) to evaluating the quality of the proposal. (If the salon members aren't omniscient at evaluating bakery quality on the merits, then they can be deceived by recommendations made for reasons other than the merits.) The salon can debate back and forth about the costs and benefits of spending the salon's snack budget at the certain bakery, but if no one present is capable of thinking "Maybe Goofusia is being nepotistic" (because anyone who could think that would never be invited to the salon), that bodes poorly for the salon's prospects of understanding the true cost–benefit landscape of catering options.
## Filtering Information Sources
>
> **Gallantina**: But then you're turning your salon into a Rev. Parris filter bubble. Don't you want your salon members to be well-read? Are you trying to save time, or are you worried about being contaminated by material that hasn't been processed and vetted by Rev. Parris?
-This one is subtle, too. If Goofusia is busy and just doesn't have time to keep up with what the world is saying about atheism and witchcraft, it might very well make sense to delegate her reading selection to Rev. Parris. That way, she can get the benefits of being mostly up to speed on these issues without having to burn too many precious hours that could be spent studying more important things.
+This one is subtle, too. If Goofusia is busy and just doesn't have time to keep up with what the world is saying about atheism and witchcraft, it might very well make sense to delegate her information gathering to Rev. Parris. That way, she can get the benefits of being mostly up to speed on these issues without having to burn too many precious hours that could be spent studying more important things.
+
+The problem is that the suggestion doesn't seem to be _about_ personal time-saving. Rev. Parris is only one person; even if he tries to make his roundups reasonably comprehensive, he can't help but omit information in ways that reflect his own biases. (For he is presumably not perfectly free of bias, and if he didn't omit anything, there would be no time-saving value to his subscribers in being able to just read the roundup rather than having to read everything that Rev. Parris reads.) If some salon members are less busy than Goofusia and can afford to do their own varied primary source reading rather than delegating it all to Rev. Parris, Goofusia should welcome that—but instead, she seems to be suspicious of it. Why?
+
+The admonition that "They do truthseeking far worse there" is a tell. The implication seems to be that good truthseekers should prefer to only read material by other good truthseekers. Rev. Parris isn't just saving his subscribers time; he's protecting them from contamination, heroically taking up the burden of extracting information out of the dangerous ravings of non-truthseekers by himself.
-The problem is that the suggestion doesn't seem to be _about_ personal time-saving. Rev. Parris is only one person; even if he tries to make his roundups reasonably comprehensive, he can't help but omit information in ways that reflect his own biases. (For he is presumably not perfectly free of bias, and if he didn't omit anything, there would be no time-saving value to his subscribers in just reading the roundup rather than having to read everything that Rev. Parris reads.) If some salon members are less busy than Goofusia and can afford to do their own varied primary source reading rather than delegating it all to Rev. Parris, Goofusia should welcome that rather than being suspicious of it.
+But it's not clear why such contamination should exist. Part of the timeless ideal of being well-read is that you're not supposed to believe everything you read. If I'm such a good truthseeker, then I should want to read everything I can about the topics I'm seeking the truth about. If the authors who publish such information aren't such good truthseekers as I am, I should take that into account when performing updates on the evidence they publish, rather than denying myself the evidence.
-The admonition that "They do truthseeking far worse there" is a tell. The implication seems to be that good truthseekers should only want to read material by other good truthseekers, and should want to avoid some sort of contamination from reading words written by worse truthseekers.
+Information is transmitted across the physical universe [through links of cause and effect](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence). If Mr. Proctor is clear-sighted and reliable, then when he reports seeing a witch, I infer that there probably was a witch. If the correlation across possible worlds is strong enough—if I think Mr. Proctor reports witches when there are witches, and not when there aren't—then Mr. Proctor's word is almost as good as if I'd seen the witch myself. If Mr. Jones has poor eyesight and is of a less reliable character, I am less credulous about reported which sightings from him, but if I don't fact any particular time constraints, I'd still rather hear Mr. Jones's testimony, because the value of information to a Bayesian reasoner is always nonnegative. Mr. Jones's report could corroborate information from other sources, for example, even if it wouldn't be definitive on its own. (Even the fact that people sometimes lie doesn't fundamentally change the calculus, [because the possibility of deception can be probabilistically "priced in"](https://www.lesswrong.com/posts/YptSN8riyXJjJ8Qp8/maybe-lying-can-t-exist).)
-But no such contamination exists. Part of the timeless ideal of being well-read is that you're not supposed to believe everything you read. Information is transmitted across the physical universe [through links of cause and effect](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence).
+That's the theory, anyway. A potential reason to fear contamination from less-truthseeking sources is that perhaps the Bayesian ideal is too hard to practice, and salon members are too prone to believe what they read. After all, many news sources have been adversarially optimized to corrupt and control their readers and make them less sane by seeing the world through ungrounded lenses.
-[TODO: reports of witch sightings can be more or less informative, but the value doesn't go zero or negative]
+But the means by which such sources manage to control their readers is precisely by capturing their trust and convincing them that they shouldn't want to read the awful corners of the internet where they do truthseeking far worse than here. Readers who have mastered _multiple_ ungrounded lenses and can check them against each other can't be owned like that. If you can spare the time, being well-read is a more robust defense against the risk of getting caught in a bad filter bubble, than trying to find a good filter bubble and blocking all (presumptively malign) outside sources of influence. All the bad bubbles have to look good from the inside, too, or they wouldn't exist.
-, because the value of information to a Bayesian reasoner is always nonnegative.
+To some, the risk of being in a bad bubble that looks good may seem too theoretical or paranoid to take seriously. It's not like there are no objective indicators of filter quality. In analogy, the observation that dreaming people don't know that they're dreaming, probably doesn't make you worry that you might be dreaming right now.
-[TODO: maybe work in "Maybe Lying Can't Exist?!", about how lies can be priced in]
+But if you are trying to be truthseeking, it being obvious that you're not in one of the worse bubbles shouldn't give you much comfort. There are still selection effects on what information gets to you, if for no other reason that there aren't enough good truthseekers to uniformly cover all the topics that a truthseeker might want to seek truth about. The sad fact is that people who write about atheism and witchcraft are disproportionately likely to be atheists or witches themselves, and therefore non-truthseeking. If your faith is so weak that you can't even risk hearing what they have to say, that necessarily limits your ability to predict and intervene on a world in which atheists and witches are real things in the physical universe that can do real harm (where you need to be able to model the things in order to figure out which interventions will reduce the harm).
## Suppressing Information Sources
-> **Goofusia**: I caught Goody Osborne distributing pamphlets quoting the honest and candid and vulnerable reflections of Rev. Samuel Parris on guiding his flock, and just trying to somehow twist that into maximum anger and hatred. It seems quite clear to me what's going on in both of the relevant pamphlets, and I think signal-boosting both of them is a pretty clear norm violation in my culture.
+> **Goofusia**: I caught Goody Osborne distributing pamphlets quoting the honest and candid and vulnerable reflections of Rev. Samuel Parris on guiding his flock, and just trying to somehow twist that into maximum anger and hatred. It seems quite clear to me what's going on in that pamphlet, and I think signal-boosting it is a pretty clear norm violation in my culture.
>
-> **Gallantina**: I read those pamphlets. They seemed like intellectually substantive satire of a public figure. If it's a norm violation to signal-boost satire of public figures,
+> **Gallantina**: I read that pamphlet. It seemed like intellectually substantive satire of a public figure. If you missed the joke, it was making fun of an alleged tendency in Rev. Parris's sermons to contain sophisticated analyses of the causes of various social ills, and then at the last moment, veer away from the uncomfortable implications and blame it all on witches. If it's a norm violation to signal-boost satire of public figures, that's artificially making it harder for people to know about flaws in the work of those public figures.
+
+This one is worse. Above, when Goofusia filtered who she talks to and what she reads for bad reasons, she was in an important sense only hurting herself. Other salon members who aren't sheltering themselves from information are unaffected by Goofusia's preference for selective ignorance, and can expect to defeat Goofusia in public debate if the need arises. The system as a whole is self-correcting.
+
+The invocation of "norm violations" changes everything. Norms depend on collective enforcement. Declaring something a norm violation is much more serious than saying that you disagree with it or don't like it; it's expressing an intent to wield [social punishment](http://zackmdavis.net/blog/2025/10/the-relationship-between-social-punishment-and-shared-maps/) in order to maintain the norm. Merely bad ideas can be criticized, but ideas that are norm-violating to signal-boost are not even to be discussed. (Discussing a work is signal-boosting it.) Norm-abiding group members are required to be ignorant of their details (or act as if they're ignorant).
+
+Mandatory ignorance of anything seems bad for truthseeking. What is Goofusia thinking here? Why would this seem like a good idea to someone?
+
+At a guess, the "maximum anger and hatred" description is load-bearing. Presumably the idea is that it's okay to calmly criticize Rev. Parris's sermons; it's only sneering or expressing anger or hatred that is forbidden. If the salon's speech code only targets form and not content, then there's no risk of the salon missing out on important content.
-This one is worse. Above, when Goofusia filtered who she talks to and what she reads, she was in an important sense only hurting herself.
+The problem is that the line between form and content is blurrier than many would prefer to believe. People don't express anger and hatred for no reason. When they do, it's because they have reasons to think something is so bad that it deserves their anger and hatred. Are they good reasons or bad reasons? If it's norm-violating to talk about it, we'll never know.
-The conjunction of "somehow" and "it seems quite clear to me" is a tell.
+Unless applied with the utmost evenhandedness and integrity, nominal censorship of form quickly morphs into censorship of content. It's an [emotive conjugation](https://en.wikipedia.org/wiki/Emotive_conjugation): I criticize; you sneer; she somehow twists into maximum anger and hatred.
+The conjunction of "somehow" and "it seems quite clear to me what's going on" is a tell.
## An Analogy to Reinforcement Learning From Human Feedback
projects / An_Algorithmic_Lucidity.git / commitdiff