From: Zack M. Davis Date: Mon, 2 Feb 2026 01:41:06 +0000 (-0800) Subject: drafting "Hazards of Selection Effects on Approved Information" X-Git-Url: https://zackmdavis.net/blog/source?a=commitdiff_plain;h=375cf4307ce276e8f955be1e50a85f3ede043972;p=An_Algorithmic_Lucidity.git drafting "Hazards of Selection Effects on Approved Information" --- diff --git a/hazards_of_selection_effects_on_approved_information.md b/hazards_of_selection_effects_on_approved_information.md new file mode 100644 index 0000000..5e3982c --- /dev/null +++ b/hazards_of_selection_effects_on_approved_information.md @@ -0,0 +1,42 @@ +# Hazards of Selection Effects on Approved Information + +In a busy, busy world, there's so much to read that no one could possibly keep up with it all. You can't _not_ prioritize what you pay attention to and (even more so) what you respond to. Everyone and her dog tells herself that she wants to pay attention to "good" (true, useful) information and ignore "bad" (false, useless) information. + +Unfortunately, there's a difference between the story everyone and her dog tells herself, and the behavior of everyone and her dog as an information-processing system in the physical universe. The story doesn't necessarily need to be true. There's nothing stopping me from telling myself a story about how I seek out true and useful information, while in fact seeking out information that _feels_ good, but which might actually be false or useless. + +Keeping the story true turns out to be a harder problem than it sounds. Everyone and her dog knows that the map is not the territory, but the reason we need a whole slogan about it is because we never actually have unmediated access to the territory. Everything we think we know about the territory is actually just part of our map (the world-simulation our brains construct from sensory data), which makes it easy to get confused about whether your actions are improving the real territory, or just your view of it on your map. + +For example, I like it when I have good ideas. It makes sense for me to like that. I endorse taking actions that will result in world-states in which I have good ideas. + +The problem is that I might not be able to tell the difference between world-states in which I have good ideas, and world-states in which I _think_ my ideas are good, but they're actually bad. Those two different states of the territory would look the same on my map. + +If my brain's learning algorithms reinforce behaviors that lead to me having ideas that I think are good, then in addition to behaviors that make me have better ideas (like reading a book), I might also inadvertently pick up behaviors that prevent me from hearing about it if my ideas are bad (like silencing critics). + +This might seem like an easy problem to solve, because the most basic manifestations of the problem are in fact pretty easy to solve. If I were to throw a crying fit and yell, "Critics bad! No one is allowed to criticize my ideas!" every time someone criticized my ideas, the problem with that would be pretty obvious to everyone and her dog, and I would stop getting invited to the salon. + +But what if there were subtler manifestations of the problem, that _weren't_ obvious to everyone and her dog? Then I might keep getting invited to the salon, and possibly even spread the problematic behavior to other salon members. (If they saw the behavior seeming to work for me, they might imitate it, and their brain's learning algorithms would reinforce it if it seemed to work for them.) What might those look like? Let's think about it. + +## Filtering Interlocutors + +> **Goofusia**: I don't see why you tolerate that distrustful, unpleasant witch Goody Osborne at your salon. Of course I understand the importance of criticism, which is an essential nutrient for any truthseeker. But you can acquire the nutrient without the downside of putting up with people like her. At least, I can. I've already got plenty of perceptive critics in my life among my friends who want the truth, and know that I want the truth. +> +> **Gallantina**: + +[TODO: "know I want the truth" is a bad filter, Parliment debating the Duke's monopoly, nutrients not homogenous] + + +## Filtering Information Sources + +> **Goofusia**: John Proctor's news makes it so I don't have to be the sort of person who reads those awful corners of the internet where these topics are discussed all day in order to understand what is happening. They do truth-seeking far worse there. +> +> **Gallantina**: + + +## Suppressing Information Sources + +> **Goofusia**: +> +> **Gallantina**: + + +## An Analogy to Reinforcement Learning From Human Feedback