The problem is that being friends with Goofusia and "know[ing] that [she and other salon members] want the truth" is a bad membership criterion, not a good one, because people who aren't friends with Goofusia and don't know that she wants the truth are likely to have different things to say. Even if Goofusia can answer all the critiques her friends can think of, that shouldn't give her confidence that her ideas are solid, if there are likely to exist serious critiques that wouldn't be independently reïnvented by the kinds of people who become Goofusia's friends.
-The "nutrient" metaphor is a tell. Goofusia seems to be thinking of criticism as if it were a homogenous ingredient necessary for a healthy epistemic environment, but that it doesn't particularly matter where it comes from. In analogy, it doesn't matter whether you get your allowance of potassium from bananas or potatoes or artificial supplements. If you find bananas and potatoes unpleasant, you can still take supplements and get your potassium that way; if you find Goody Osborne unpleasant, you can just talk to your friends who know you want the truth and get your criticism that way.
+The "nutrient" metaphor is a tell. Goofusia seems to be thinking of criticism as if it were a homogeneous ingredient necessary for a healthy epistemic environment, but that it doesn't particularly matter where it comes from. In analogy, it doesn't matter whether you get your allowance of potassium from bananas or potatoes or artificial supplements. If you find bananas and potatoes unpleasant, you can still take supplements and get your potassium that way; if you find Goody Osborne unpleasant, you can just talk to your friends who know you want the truth and get your criticism that way.
-But unlike chemically uniform nutrients, criticism isn't homogenous: different critics are differently equipped by virtue of their different intellectual backgrounds to notice different flaws in a piece of work. The purpose of criticism is not to virtuously endure being criticized; the purpose is to surface and fix every individual flaw. (If you independently got everything exactly right the first time, then there would be nothing for critics to do; that just seems pretty unlikely if you're talking about anything remotely complicated. It would be hard to believe that such an unlikely-seeming thing had really happened without the best critics getting the opportunity to do their worst.)
+But unlike chemically uniform nutrients, criticism isn't homogeneous: different critics are differently equipped by virtue of their different intellectual backgrounds to notice different flaws in a piece of work. The purpose of criticism is not to virtuously endure being criticized; the purpose is to surface and fix every individual flaw. (If you independently got everything exactly right the first time, then there would be nothing for critics to do; that just seems pretty unlikely if you're talking about anything remotely complicated. It would be hard to believe that such an unlikely-seeming thing had really happened without the best critics getting the opportunity to do their worst.)
"Knowing that (someone) wants the truth" is a particularly poor filter, because people who think that they have strong criticisms of your ideas [are particularly likely to think that you don't want the truth](https://www.lesswrong.com/posts/iThwqe3yPog56ytyq/aiming-for-convergence-is-like-discouraging-betting). (Because, the reasoning would go, if you did want the truth, why would you propose such flawed ideas, instead of independently inventing the obvious-to-them criticism yourself and dropping the idea without telling anyone?) Refusing to talk to people who think that they have strong criticisms of your ideas is a bad thing to do if you care about your ideas being correct.
To some, the risk of being in a bad bubble that looks good may seem too theoretical or paranoid to take seriously. It's not like there are no objective indicators of filter quality. In analogy, the observation that dreaming people don't know that they're dreaming, probably doesn't make you worry that you might be dreaming right now.
-But if you are trying to be truthseeking, it being obvious that you're not in one of the worse bubbles shouldn't give you much comfort. There are still selection effects on what information gets to you, if for no other reason that there aren't enough good truthseekers to uniformly cover all the topics that a truthseeker might want to seek truth about. The sad fact is that people who write about atheism and witchcraft are disproportionately likely to be atheists or witches themselves, and therefore non-truthseeking. If your faith is so weak that you can't even risk hearing what they have to say, that necessarily limits your ability to predict and intervene on a world in which atheists and witches are real things in the physical universe that can do real harm (where you need to be able to model the things in order to figure out which interventions will reduce the harm).
+But if you are trying to be truthseeking, it being obvious that you're not in one of the worse bubbles shouldn't give you much comfort. There are still selection effects on what information gets to you, if for no other reason that there aren't enough good truthseekers to uniformly cover all the topics that a truthseeker might want to seek truth about. The sad fact is that people who write about atheism and witchcraft are disproportionately likely to be atheists or witches themselves, and therefore non-truthseeking. If your faith in truthseeking is so weak that you can't even risk hearing what non-truthseekers have to say, that necessarily limits your ability to predict and intervene on a world in which atheists and witches are real things in the physical universe that can do real harm (where you need to be able to model the things in order to figure out which interventions will reduce the harm).
## Suppressing Information Sources
This one is worse. Above, when Goofusia filtered who she talks to and what she reads for bad reasons, she was in an important sense only hurting herself. Other salon members who aren't sheltering themselves from information are unaffected by Goofusia's preference for selective ignorance, and can expect to defeat Goofusia in public debate if the need arises. The system as a whole is self-correcting.
-The invocation of "norm violations" changes everything. Norms depend on collective enforcement. Declaring something a norm violation is much more serious than saying that you disagree with it or don't like it; it's expressing an intent to wield [social punishment](http://zackmdavis.net/blog/2025/10/the-relationship-between-social-punishment-and-shared-maps/) in order to maintain the norm. Merely bad ideas can be criticized, but ideas that are norm-violating to signal-boost are not even to be discussed. (Discussing a work is signal-boosting it.) Norm-abiding group members are required to be ignorant of their details (or act as if they're ignorant).
+The invocation of "norm violations" changes everything. Norms depend on collective enforcement. Declaring something a norm violation is much more serious than saying that you disagree with it or don't like it; it's expressing an intent to wield [social punishment](http://zackmdavis.net/blog/2025/10/the-relationship-between-social-punishment-and-shared-maps/) in order to maintain the norm. Merely bad ideas can be criticized, but ideas that are norm-violating to signal-boost are presumably not even to be discussed. (Discussing a work is signal-boosting it.) Norm-abiding group members are required to be ignorant of their details (or act as if they're ignorant).
Mandatory ignorance of anything seems bad for truthseeking. What is Goofusia thinking here? Why would this seem like a good idea to someone?
-At a guess, the "maximum anger and hatred" description is load-bearing. Presumably the idea is that it's okay to calmly criticize Rev. Parris's sermons; it's only sneering or expressing anger or hatred that is forbidden. If the salon's speech code only targets form and not content, then there's no risk of the salon missing out on important content.
+At a guess, the "maximum anger and hatred" description is load-bearing. Presumably the idea is that it's okay to calmly criticize Rev. Parris's sermons; it's only sneering or expressing anger or hatred that is forbidden. If the salon's speech code only targets form and not content, the reasoning goes, then there's no risk of the salon missing out on important content.
-The problem is that the line between form and content is blurrier than many would prefer to believe. People don't express anger and hatred for no reason. When they do, it's because they have reasons to think something is so bad that it deserves their anger and hatred. Are they good reasons or bad reasons? If it's norm-violating to talk about it, we'll never know.
+The problem is that the line between form and content is blurrier than many would prefer to believe, because words mean things. You can't just swap in non-angry words for angry words without changing the meaning of a sentence. Maybe the distortion of meaning introduced by substituting the nicer words is small, but then again, maybe it's large: the only person in a position to say is the author. People don't express anger and hatred for no reason. When they do, it's because they have reasons to think something is so bad that it deserves their anger and hatred. Are those good reasons or bad reasons? If it's norm-violating to talk about it, we'll never know.
-Unless applied with the utmost evenhandedness and integrity, nominal censorship of form quickly morphs into censorship of content. It's an [emotive conjugation](https://en.wikipedia.org/wiki/Emotive_conjugation): I criticize; you sneer; she somehow twists into maximum anger and hatred.
+Unless applied with the utmost stringent evenhandedness and integrity, censorship of form quickly morphs into censorship of content, as heated criticism of the ingroup is construed as norm-violating, while equally heated criticism of the outgroup is unremarkable and gets a pass. It's an [emotive conjugation](https://en.wikipedia.org/wiki/Emotive_conjugation): I criticize; you sneer; she somehow twists into maximum anger and hatred.
-The conjunction of "somehow" and "it seems quite clear to me what's going on" is a tell.
+The conjunction of "somehow" and "it seems quite clear to me what's going on" is a tell. If it were _actually_ clear to Goofusia what was going on with the pamphlet author expressing anger and hatred towards Rev. Parris, she would not use the word "somehow" to describe the author's behavior: she would be able to pass the author's ideological Turing test and therefore know exactly how.
+
+If that were just Goofusia's mistake, the loss would be hers alone, but if Goofusia is in a position of social power over others, she might succeed at spreading her anti-speech, anti-reading cultural practices to others. I can only imagine that the result would be a subculture that was obsessively self-congratulatory about its own superiority in "truthseeking", while simultaneously blind to everything outside itself. People spending their lives immersed in that culture wouldn't necessarily notice anything was wrong from the inside. What could you say to help them?
## An Analogy to Reinforcement Learning From Human Feedback
-[TODO: you can't "just not" do RLHF and still have a competitive AI in the current paradigm, but you can try to be aware of problems with supervision; it's better if the CEO says, "Yes, we thought of that, too; we think it's fine because of these-and-such mitigations." If the CEO says, "Well, _I_ think our raters are great. Are you insulting our raters?", that does not inspire confidence
-]
+Pointing out problems is easy. Finding solutions is harder.
+
+The training pipeline for frontier language models typically includes a step called reinforcement learning from human feedback (RLHF). Before that, previous stages of training have already produced a "base" language model that predicts continuations of internet text, but base models aren't very useful: we want AIs that answer our questions and obey our commands, not just predict what token would come next in internet text. The solution is RLHF: a reward model (often just a language model with a different final layer) is trained to predict the judgments of human raters about which of a pair of continuations is better (for the purposes of what the humans want out of an AI, like answering questions), and the base model is optimized against the reward model.
+
+The problem is that human feedback (and the reward model's prediction of it) is imperfect. The reward model [can't tell the difference](https://www.lesswrong.com/posts/xFotXGEotcKouifky/worlds-where-iterative-design-fails) between "The AI is being good" and "The AI looks good to the reward model". This already has the failure mode of sycophancy, in which today's large language models tell users what they want to hear, but it's been theorized that much larger harms (up to and including human extinction) could materialize from future AI systems doing what we trained them to do rather than what we _think_ we trained them to do.
+
+The problem doesn't have any simple, obvious solution. In the absence of some sort of international treaty to halt all AI development worldwide, "Just don't do RLHF" isn't a feasible and doesn't even make any sense; you need some sort of feedback in order to make an AI that does anything useful at all. The problem may or may not ultimately be solvable, but the first step on the road to some future complicated, nonobvious solution is acknowledging that the the problem is at least potentially real and having some respect that the problem might be difficult.
+
+If a safety auditor comes to the CEO expressing concerns about the company's RLHF pipeline being unsafe due to imperfect feedback, it's more reassuring if the CEO says, "Yes, we thought of that, too; we've implemented these-and-such mitigations, and are monitoring such-and-these signals, which we hope will clue us in if the mitigations start to fail." If the CEO instead says, "Well, _I_ think our raters are great. Are you insulting our raters?", that does not inspire confidence. The natural inference is that the CEO is mostly interested in this quarter's profits and doesn't really care about safety.
+
+Similarly, the problem with your truthseeking community being imperfect and not being able to tell the difference between "We have the right answers" and "We have what we think are the right answers" doesn't have any simple, obvious solution. "Just don't filter information" isn't feasible and doesn't even make any sense; you need some sort of filter because it's not physically possible to read everything and respond to everything.
+
+But the first step on the road to some future complicated, nonobvious solution is acknowledging that the the problem is at least potentially real and having some respect that the problem might be difficult. If community leaders respond to concerns about collective belief distortions due to suppression of criticism with silence, "bowing out", or defensive blustering, rather than "Yes, we thought of that, too", that does not inspire confidence. The natural inference is that they're mostly interested in maintaining the group's reputation and don't really care about truth.
projects / An_Algorithmic_Lucidity.git / commitdiff