From 01fe0979b524a172f1836bb207915fdb36b06acb Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Thu, 5 Feb 2026 19:39:07 -0800 Subject: [PATCH] drafting "Hazards of Selection Effects on Approved Information" --- ...lection_effects_on_approved_information.md | 31 ++++++++++++------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/hazards_of_selection_effects_on_approved_information.md b/hazards_of_selection_effects_on_approved_information.md index c250d23..9c1499d 100644 --- a/hazards_of_selection_effects_on_approved_information.md +++ b/hazards_of_selection_effects_on_approved_information.md @@ -24,32 +24,40 @@ But what if there were subtler manifestations of the problem, that _weren't_ obv This one is subtle. Goofusia isn't throwing a crying fit every time a member of the salon criticizes her ideas. And indeed, you can't invite the whole world to your salon. You can't _not_ do some sort of filtering. The question is whether salon invitations are being extended or withheld for "good" reasons (that promote the salon processing true and useful information) or "bad" reasons (that promote false or useless information). -The problem is that being friends with Goofusia and "know[ing] that [she and other salon members] want the truth" is a bad membership criterion, not a good one, because people who aren't friends with Goofusia and don't know that she wants the truth are likely to have different things to say. Even if Goofusia can answer all the critiques her friends can think of, that shouldn't give her confidence that her ideas are solid if there are likely to be possible serious critiques that wouldn't be independently generated by the kinds of people who become Goofusia's friends. +The problem is that being friends with Goofusia and "know[ing] that [she and other salon members] want the truth" is a bad membership criterion, not a good one, because people who aren't friends with Goofusia and don't know that she wants the truth are likely to have different things to say. Even if Goofusia can answer all the critiques her friends can think of, that shouldn't give her confidence that her ideas are solid, if there are likely to be serious critiques that wouldn't be independently reïnvented by the kinds of people who become Goofusia's friends. -The "nutrient" metaphor is a tell. Goofusia seems to be thinking of criticism as if it were a homogenous ingredient necessary for a healthy epistemic environment, but that it doesn't particularly matter where it comes from. In analogy, it doesn't matter whether you get your allowance of potassium from bananas or potatoes or artificial supplements. If you find bananas and potatoes unpleasant, you can still take supplements and get the potassium that way. +The "nutrient" metaphor is a tell. Goofusia seems to be thinking of criticism as if it were a homogenous ingredient necessary for a healthy epistemic environment, but that it doesn't particularly matter where it comes from. In analogy, it doesn't matter whether you get your allowance of potassium from bananas or potatoes or artificial supplements. If you find bananas and potatoes unpleasant, you can still take supplements and get your potassium that way. -But unlike chemically uniform nutrients, criticism isn't homogenous: different critics are differently equipped to notice different flaws in a piece of work. +But unlike chemically uniform nutrients, criticism isn't homogenous: different critics are differently equipped by virtue of their different intellectual backgrounds to notice different flaws in a piece of work. The purpose of criticism is not to virtuously endure being criticized; the purpose is to surface and fix every individual flaw. +"Knowing that (someone) wants the truth" is a particularly poor filter, because people who think that they have strong criticisms of your ideas are particularly likely to think that you don't want the truth. (Because, the reasoning would go, [if you did want the truth, why would you propose such flawed ideas, instead of independently inventing the obvious-to-them criticism yourself](https://www.lesswrong.com/posts/iThwqe3yPog56ytyq/aiming-for-convergence-is-like-discouraging-betting) and dropping the idea without telling anyone?) Refusing to talk to people who think that they have strong criticisms of your ideas is a bad thing to do if you care about your ideas being correct. -[TODO: "know I want the truth" is a bad filter, salon bakery] - +The effect is particularly bad in situations where the fact that someone doesn't want the truth is relevant to the correct answer. Suppose Goofusia proposes that the salon buys cookies from a certain bakery—which happens to be owned by Goofusia's niece. If Goofusia's proposal was motivated by nepotism, that's [probabilistically relevant](https://www.lesswrong.com/posts/y4bkJTtG3s5d6v36k/stupidity-and-dishonesty-explain-each-other-away) to evaluating the quality of the proposal. (If the salon members aren't omniscient at evaluating bakery quality on the merits, then they can be deceived by recommendations made for reasons other than the merits.) The salon can debate back and forth about the costs and benefits of spending the salon's snack budget at the certain bakery, but if no one present is capable of thinking "Maybe Goofusia is being nepotistic" (because anyone who could think that would never be invited to the salon), that bodes poorly for the salon's prospects of understanding the true cost–benefit landscape of catering options. ## Filtering Information Sources -> **Goofusia**: One shouldn't have to be the sort of person who follows discourse in crappy filter-bubbles in order to understand what's happening. The Rev. Samuel Parris's news summaries are the sort of thing that lets me do that. Our salon should work like that if it's going to talk about the atheist threat and the witchcraft crisis. I don't want to have to read the awful corners of the internet where this is discussed all day. They do truthseeking far worse there. +> **Goofusia**: One shouldn't have to be the sort of person who follows discourse in crappy filter-bubbles in order to understand what's happening. The Rev. Samuel Parris's news summary roundups are the sort of thing that lets me do that. Our salon should work like that if it's going to talk about the atheist threat and the witchcraft crisis. I don't want to have to read the awful corners of the internet where this is discussed all day. They do truthseeking far worse there. > -> **Gallantina**: But then you're turning your salon into a Rev. Parris filter bubble. Don't you want your salon members to be well-read? [TODO: followup question alluding to contamination frame] +> **Gallantina**: But then you're turning your salon into a Rev. Parris filter bubble. Don't you want your salon members to be well-read? Are you trying to save time, or are you worried about being contaminated by material that hasn't been processed and vetted by Rev. Parris? + +This one is subtle, too. If Goofusia is busy and just doesn't have time to keep up with what the world is saying about atheism and witchcraft, it might very well make sense to delegate her reading selection to Rev. Parris. That way, she can get the benefits of being mostly up to speed on these issues without having to burn too many precious hours that could be spent studying more important things. + +The problem is that the suggestion doesn't seem to be _about_ personal time-saving. Rev. Parris is only one person; even if he tries to make his roundups reasonably comprehensive, he can't help but omit information in ways that reflect his own biases. (For he is presumably not perfectly free of bias, and if he didn't omit anything, there would be no time-saving value to his subscribers in just reading the roundup rather than having to read everything that Rev. Parris reads.) If some salon members are less busy than Goofusia and can afford to do their own varied primary source reading rather than delegating it all to Rev. Parris, Goofusia should welcome that rather than being suspicious of it. -This one is similar. +The admonition that "They do truthseeking far worse there" is a tell. The implication seems to be that good truthseekers should only want to read material by other good truthseekers, and should want to avoid some sort of contamination from reading words written by worse truthseekers. -The "awful", "crappy", and "do[ing] truthseeking far worse there" characterizations are a tell. +But no such contamination exists. Part of the timeless ideal of being well-read is that you're not supposed to believe everything you read. Information is transmitted across the physical universe [through links of cause and effect](https://www.lesswrong.com/posts/6s3xABaXKPdFwA3FS/what-is-evidence). -[TODO: filtering interlocutors is more forgiveable if they're expensive, but reading contamination is wack] +[TODO: reports of witch sightings can be more or less informative, but the value doesn't go zero or negative] + +, because the value of information to a Bayesian reasoner is always nonnegative. + +[TODO: maybe work in "Maybe Lying Can't Exist?!", about how lies can be priced in] ## Suppressing Information Sources -> **Goofusia**: I caught Goody Osborne distributing pamphlets quoting the honest and candid and vulnerable reflections of Rev. Samuel Parris on guiding his flock, in pretty obviously reasonable ways, and just trying to somehow twist that into maximum anger and hatred. It seems quite clear to me what's going on in both of the relevant pamphlets, and I think signal boosting both of them is a pretty clear norm violation in my culture. +> **Goofusia**: I caught Goody Osborne distributing pamphlets quoting the honest and candid and vulnerable reflections of Rev. Samuel Parris on guiding his flock, and just trying to somehow twist that into maximum anger and hatred. It seems quite clear to me what's going on in both of the relevant pamphlets, and I think signal-boosting both of them is a pretty clear norm violation in my culture. > > **Gallantina**: I read those pamphlets. They seemed like intellectually substantive satire of a public figure. If it's a norm violation to signal-boost satire of public figures, @@ -58,6 +66,7 @@ This one is worse. Above, when Goofusia filtered who she talks to and what she r The conjunction of "somehow" and "it seems quite clear to me" is a tell. + ## An Analogy to Reinforcement Learning From Human Feedback [TODO: you can't "just not" do RLHF and still have a competitive AI in the current paradigm, but you can try to be aware of problems with supervision; it's better if the CEO says, "Yes, we thought of that, too; we think it's fine because of these-and-such mitigations." If the CEO says, "Well, _I_ think our raters are great. Are you insulting our raters?", that does not inspire confidence -- 2.53.0