An Algorithmic Lucidity

(originally published at Less Wrong)

As a casual formal epistemology fan, you've probably heard that the philosophical notion of causality can be formalized in terms of Bayesian networks—but also as a casual formal epistemology fan, you also probably don't know the details all that well.

One day, while going through the family archives, you come across a meticulously maintained dataset describing a joint probability distribution over four variables: whether it rained that day, whether the sprinkler was on, whether the sidewalk was wet, and whether the sidewalk was slippery. The distribution is specified in this table (using the abbreviated labels "rain", "slippery", "sprinkler", and "wet"):

(You wonder what happened that one day out of 140,000 when it rained, and the sprinkler was on, and the sidewalk was slippery but not wet. Did—did someone put a tarp up to keep the sidewalk dry, but also spill slippery oil, which didn't count as being relevantly "wet"? Also, 140,000 days is more than 383 years—were "sprinklers" even a thing in the year 1640 C.E.? You quickly put these questions out of your mind: it is not your place to question the correctness of the family archives.)

You're slightly uncomfortable with this unwieldy sixteen-row table. You think that there must be some other way to represent the same information, while making it clearer that it's not a coincidence that rain and wet sidewalks tend to co-occur.

You've read that Bayesian networks "factorize" an unwieldly joint probability distribution into a number of more compact conditional probability distributions, related by a directed acyclic graph, where the arrows point from "cause" to "effect". (Even a casual formal epistemology fan knows that much.) The graph represents knowledge that each variable is conditionally independent of its non-descendants given its parents, which enables "local" computations: given the values of just a variable's parents in the graph, we can compute a conditional distribution for that variable, without needing to consider what is known about other variables elsewhere in the graph ...

You've read that, but you've never actually done it before! You decide that constructing a Bayesian network to represent this distribution will be a useful exercise.

To start, you re-label the variables for brevity. (On a whim, you assign indices in reverse-alphabetical order: \(X_1\) = wet, \(X_2\) = sprinkler, \(X_3\) = slippery, \(X_4\) = rain.)

Now, how do you go about building a Bayesian network? As a casual formal epistemology fan, you are proud to own a copy of the book by Daphne Koller and the other guy, which explains how to do this in—you leaf through the pages—probably §3.4, "From Distributions to Graphs"?—looks like ... here, in Algorithm 3.2. It says to start with an empty graph, and it talks about random variables, and setting directed edges in the graph, and you know from chapter 2 that the ⟂ and | characters are used to indicate conditional independence. That has to be it.

(As a casual formal epistemology fan, you haven't actually read chapter 3 up through §3.4, but you don't see why that would be necessary, since this Algorithm 3.2 pseudocode is telling you what you need to do.)

It looks like the algorithm says to pick a variable, allocate a graph node to represent it, find the smallest subset of the previously-allocated variables such that the variable represented by the new node is conditionally independent of the other previously-allocated variables given that subset, and then draw directed edges from each of the nodes in the subset to the new node?—and keep doing that for each variable—and then compute conditional probability tables for each variable given its parents in the resulting graph?

That seems complicated when you say it abstractly, but you have faith that it will make more sense as you carry out the computations.

First, you allocate a graph node for \(X_1\). It doesn't have any parents, so the associated conditional ("conditional") probability distribution, is really just the marginal distribution for \(X_1\).

Then you allocate a node for \(X_2\). \(X_2\) is not independent of \(X_1\). (Because \(P(X_1 \land X_2)\) = 169/1400, which isn't the same as \(P(X_1) \cdot P(X_2)\) = 8/25 · 1/7 = 8/175.) So you make \(X_1\) a parent of \(X_2\), and your conditional probability table for \(X_2\) separately specifies the probabilities of \(X_2\) being true or false, depending on whether \(X_1\) is true or false.

Next is \(X_3\). Now that you have two possible parents, you need to check whether conditioning on either of \(X_1\) and \(X_2\) would render \(X_3\) conditionally independent of the other. If not, then both \(X_1\) and \(X_2\) will be parents of \(X_3\); if so, then the variable you conditioned on will be the sole parent. (You assume that the case where \(X_3\) is just independent from both \(X_1\) and \(X_2\) does not pertain; if that were true, \(X_3\) wouldn't be connected to the rest of the graph at all.)

It turns out that \(X_3\) and \(X_2\) are conditionally independent given \(X_1\). That is, \(P(X_3 \land X_2 \mid X_1) = P(X_3 \mid X_1) \cdot P(X_2 \mid X_1)\). (Because the left-hand side is \(\frac{P(X_3 \land X_2 \land X_1)}{P(X_1)} = \frac{507}{1792}\), and the right-hand side is \(\frac{3}{4} \cdot \frac{169}{448} = \frac{507}{1792}\).) So \(X_1\) is a parent of \(X_3\), and \(X_2\) isn't; you draw an arrow from \(X_1\) (and only \(X_1\)) to \(X_3\), and compile the corresponding conditional probability table.

Finally, you have \(X_4\). The chore of finding the parents is starting to feel more intuitive now. Out of the \(2^3 = 8\) possible subsets of the preceding variables, you need to find the smallest subset, such that conditioning on that subset renders \(X_4\) (conditionally) independent of the variables not in that subset. After some calculations that the authors of expository blog posts have sometimes been known to callously leave as an exercise to the reader, you determine that \(X_1\) and \(X_2\) are the parents of \(X_4\).

And with one more conditional probability table, your Bayesian network is complete!

Eager to interpret the meaning of this structure regarding the philosophy of causality, you translate the \(X_i\) variable labels back to English:

...

This can't be right. The arrow from "wet" to "slippery" seems fine. But all the others are clearly absurd. Wet sidewalks cause rain? Sprinklers cause rain? Wet sidewalks cause the sprinkler to be on?

You despair. You thought you had understood the algorithm. You can't find any errors in your calculations—but surely there must be some? What did you do wrong?

After some thought, it becomes clear that it wasn't just a calculation error: the procedure you were trying to carry out couldn't have given you the result you expected, because it never draws arrows from later-considered to earlier-considered variables. You considered "wet" first. You considered "rain" last, and then did independence tests to decide whether or not to draw arrows from "wet" (or "sprinkler" or "slippery") to "rain". An arrow from "rain" to "wet" was never a possibility. The output of the algorithm is sensitive to the ordering of the variables.

(In retrospect, that probably explains the "given an ordering" part of Algorithm 3.2's title, "Procedure to build a minimal I-map given an ordering." You hadn't read up through the part of chapter 3 that presumably explains what an "I-map" is, and had disregarded the title as probably unimportant.)

You try carrying out the algorithm with the ordering "rain", "sprinkler", "wet", "slippery" (or \(X_4\), \(X_2\), \(X_1\), \(X_3\) using your \(X_i\) labels from before), and get this network:

—for which giving the arrows a causal interpretation seems much more reasonable.

You notice that you are very confused. The "crazy" network you originally derived, and this "true" network derived from a more intuitively causal variable ordering, are different: they don't have the same structure, and (except for the wet → slippery link) they don't have the same conditional probability tables. You would assume that they can't "both be right". If the network output by the algorithm depends on what variable ordering you use, how are you supposed to know which ordering is correct? In this example, you know from reasons outside the math, that "wet" shouldn't cause "rain", but you couldn't count on that were you to apply these methods to problems further removed from intuition.

Playing with both networks, you discover that despite their different appearances, they both seem to give the same results when you use them to calculate marginal or conditional probabilities. For example, in the "true" network, \(P(\mathrm{rain})\) is 1/4 (read directly from the "conditional" probability table, as "rain" has no parents in the graph). In the "crazy" network, the probability of rain can be computed as

... which also equals 1/4.

That actually makes sense. You were wrong to suppose that the two networks couldn't "both be right". They are both right; they both represent the same joint distribution. The result of the algorithm for constructing a Bayesian network—or a "minimal I-map", whatever that is—depends on the given variable ordering, but since the algorithm is valid, each of the different possible results is also valid.

But if the "crazy" network and the "true" network are both right, what happened to the promise of understanding causality using Bayesian networks?! (You may only be a casual formal epistemology fan, but you remember reading a variety of secondary sources unanimously agreeing that this was a thing; you're definitely not misremembering or making it up.) If both networks give the same answers to marginal and conditional probability queries, that amounts to them making the same predictions about the world. So if beliefs are supposed to correspond to predictions, in what sense could the "true" network be better? What does your conviction that rain causes wetness even mean, if someone who believed the opposite could make all the same predictions?

You remember the secondary sources talking about interventions on causal graphs: severing a node from its parents and forcing it to take a particular value. And the "crazy" network and the "true" network do differ with respect to that operation: in the "true" network, setting "wet" to be false—you again imagine putting a tarp up over the sidewalk—wouldn't change the probability of "rain". But in the "crazy" network, forcing "wet" to be false would change the probability of rain—to \(P(\mathrm{rain} \mid \mathrm{sprinkler}, \neg \mathrm{wet}) \cdot P(\mathrm{sprinkler} \mid \neg \mathrm{wet}) + P(\mathrm{rain} \mid \neg \mathrm{sprinkler}, \neg \mathrm{wet}) \cdot P(\neg \mathrm{sprinkler} \mid \neg \mathrm{wet})\), which is \(\frac{1}{31} \cdot \frac{31}{952} + \frac{10}{307} \cdot \frac{921}{952} \approx 0.032\) (greatly reduced from the 1/4 you calculated a moment ago). Notably, this intervention—\(P(\mathrm{rain} \mid \mathrm{do}(\neg \mathrm{wet}))\), if you're remembering correctly what some of the secondary sources said about a do operator—isn't the same thing as the conditional probability \(P(\mathrm{rain}| \neg \mathrm{wet})\).

This would seem to satisfy your need for a sense in which the "true" network is "better" than the "crazy" network, even if Algorithm 3.2 indifferently produces either depending on the ordering it was given. (You're sure that Daphne Koller and the other guy have more to say about other algorithms that can make finer distinctions, but this feels like enough studying for one day—and enough for one expository blog post, if someone was writing one about your inquiries. You're a casual formal epistemology fan.) The two networks represent the same predictions about the world recorded in your family archives, but starkly different predictions about nearby possible worlds—about what would happen if some of the factors underlying the world were to change.

You feel a slight philosophical discomfort about this. You don't like the idea of forced change, of intervention, being so integral to such a seemingly basic notion as causality. It feels almost anthropomorphic: you want the notion of cause and effect within a system to make sense without reference to the intervention of some outside agent—for there's nothing outside of the universe. But whether this intuition is a clue towards deeper insights, or just a place where your brain has tripped on itself and gotten confused, it's more than you understand now.

(originally published at Less Wrong)

Derek Powell, Kara Weisman, and Ellen M. Markman's "Articulating Lay Theories Through Graphical Models: A Study of Beliefs Surrounding Vaccination Decisions" (a conference paper from CogSci 2018) represents an exciting advance in marketing research, showing how to use causal graphical models to study why ordinary people have the beliefs they do, and how to intervene to make them be less wrong.

The specific case our authors examine is that of childhood vaccination decisions: some parents don't give their babies the recommended vaccines, because they're afraid that vaccines cause autism. (Not true.) This is pretty bad—not only are those unvaccinated kids more likely to get sick themselves, but declining vaccination rates undermine the population's herd immunity, leading to new outbreaks of highly-contagious diseases like the measles in regions where they were once eradicated.

What's wrong with these parents, huh?! But that doesn't have to just be a rhetorical question—Powell et al. show how we can use statistics to make the rhetorical hypophorical and model specifically what's wrong with these people! Realistically, people aren't going to just have a raw, "atomic" dislike of vaccination for no reason: parents who refuse to vaccinate their children do so because they're (irrationally) afraid of giving their kids autism, and not afraid enough of letting their kids get infectious diseases. Nor are beliefs about vaccine effectiveness or side-effects uncaused, but instead depend on other beliefs.

To unravel the structure of the web of beliefs, our authors got Amazon Mechanical Turk participants to take surveys about vaccination-related beliefs, rating statements like "Natural things are always better than synthetic alternatives" or "Parents should trust a doctor's advice even if it goes against their intuitions" on a 7-point Likert-like scale from "Strongly Agree" to "Strongly Disagree".

Throwing some off-the-shelf Bayes-net structure-learning software at a training set from the survey data, plus some ancillary assumptions (more-general "theory" beliefs like "skepticism of medical authorities" can cause more-specific "claim" beliefs like "vaccines have harmful additives", but not vice versa) produces a range of probabilistic models that can be depicted with graphs where nodes representing the different beliefs are connected by arrows that show which beliefs "cause" others: an arrow from a naturalism node (in this context, denoting a worldview that prefers natural over synthetic things) to a parental expertise node means that people think parents know best because they think that nature is good, not the other way around.

Learning these kinds of models is feasible because not all possible causal relationships are consistent with the data: if \(A\) and \(B\) are statistically independent of each other, but each dependent with \(C\) (and are conditionally dependent given the value of \(C\)), it's kind of hard to make sense of this except to posit that \(A\) and \(B\) are causes with the common effect \(C\).

Simpler models with fewer arrows might sacrifice a little bit of predictive accuracy for the benefit of being more intelligible to humans. Powell et al. ended up choosing a model that can predict responses from the test set at r = .825, explaining 68.1% of the variance. Not bad?!—check out the full 14-node graph in Figure 2 on page 4 of the PDF.

Causal graphs are useful as a guide for planning interventions: the graph encodes predictions about what would happen if you changed some of the variables. Our authors point out that since previous work showed that people's beliefs about vaccine dangers were difficult to influence, that suggests trying to intervene on the other parents of the intent-to-vaccinate node in the model: if the hoi polloi won't listen to you when you tell them the costs are minimal (vaccines are safe), instead tell them about the benefits (diseases are really bad and vaccines prevent disease).

To make sure I really understand this, I want to adapt it into a simpler example with made-up numbers where I can do the arithmetic myself. Let me consider a graph with just three nodes—

Suppose this represents a structural equation model where an anti-vaxxer-leaning parent-to-be's propensity-to-vaccinate-against-measles \(C\) is expressed in terms of belief-in-vaccine-safety \(A\) and belief-in-measles-danger \(B\) as—

And suppose that we're a public health authority trying to decide whether to spend our budget (or what's left of it after recent funding cuts) on a public education initiative that will increase \(A\) by 0.1, or one that will increase \(B\) by 0.3.

We should choose the program that intervenes on \(B\), because \((0.3)(0.3) = 0.09\) is bigger than \((0.7)(0.1) = 0.07\). That's actionable advice that we couldn't have derived without a quantitative model of how the lay audience thinks. Exciting!

At this point, some readers may be wondering why I've described this work as "marketing research" about constructing "optimized propaganda." A couple of those words usually have negative connotations, but educating people about the importance of vaccines is a positive thing. What gives?

The thing is, "Learn the causal graph of why they think that and compute how to intervene on it to make them think something else" is a symmetric weapon—a fully general persuasive technique that doesn't depend on whether the thing you're trying to convince them of is true.

In my simplified example, the choice to intervene on \(B\) was based on numerical assumptions that amount to the claim that it's sufficiently easier to change \(B\) than it is to change \(A\), such that intervening on \(B\) is more effective at changing \(C\) than intervening on \(A\) (even though \(C\) depends on \(A\) more than it does on \(B\)). But this methodology is completely indifferent to what \(A\), \(B\), and \(C\) mean. It would have worked just as well, and for the same reasons if the graph had been—

Suppose that we're advertising executives for the Coca-Cola Company trying to decide how to spend our budget (or what's left of it after recent funding cuts). If consumers won't listen to us when we tell them the costs of drinking Coke are minimal (lying that it isn't unhealthy), we should instead tell them about the benefits (Coke tastes good).

Or with different assumptions about the parameters—maybe \(C = 0.8 \cdot A + 0.2 \cdot B\) actually—then intervening to increase belief in "Coca-Cola isn't unhealthy" would be the right move (because \((0.8)(0.1) = 0.08 > 0.06 = (0.2)(0.3)\)). The marketing algorithm that just computes what belief changes will flip the decision node, doesn't have any way to notice or care whether those belief changes are in the direction of more or less accuracy.

To be clear—and I really shouldn't have to say this—this is not a criticism of Powell–Weisman–Markman's research! The "Learn the causal graph of why they think that" methodology is genuinely really cool! It doesn't have to be deployed as a marketing algorithm: the process of figuring out which belief change would flip some downstream node is the same thing as what we call locating a crux.¹ The difference is just a matter of forwards or backwards direction: whether you first figure out if the measles vaccine or Coca-Cola are safe and then use whatever answer you come up with to guide your decision, or whether you write the bottom line first.

Of course, most people on most issues don't have the time or expertise to do their own research. For the most part, we can only hope that the sources we trust as authorities are doing their best to use their limited bandwidth to keep us genuinely informed, rather than merely computing what signals to emit in order to control our decisions.

If that's not true, we might be in trouble—perhaps increasingly so, if technological developments grant new advantages to the propagation of disinformation over the discernment of truth. In a possible future world where most words are produced by AIs running a "Learn the causal graph of why they think that and intervene on it to make them think something else" algorithm hooked up to a next-generation GPT, even reading plain text from an untrusted source could be dangerous.

(originally published at Less Wrong)

Sometimes, people are reluctant to admit that they were wrong about something, because they're afraid that "You are wrong about this" carries inextricable connotations of "You are stupid and bad." But this behavior is, itself, wrong, for at least two reasons.

First, because it's evidential decision theory. The so-called "rationalist" "community" has a lot of cached clichés about this! A blank map does not correspond to a blank territory. What's true is already so; owning up to it doesn't make it worse. Refusing to go to the doctor (thereby avoiding encountering evidence that you're sick) doesn't keep you healthy.

If being wrong means that you're stupid and bad, then preventing yourself from knowing that you were wrong doesn't stop you from being stupid and bad in reality. It just prevents you from knowing that you're stupid and bad—which is an important fact to know (if it's true), because if you don't know that you're stupid and bad, then it probably won't occur to you to even look for possible interventions to make yourself less stupid and less bad.

Second, while "You are wrong about this" is evidence for the "You are stupid and bad" hypothesis if stupid and bad people are more likely to be wrong, I claim that it's very weak evidence. (Although it's possible that I'm wrong about this—and if I'm wrong, it's furthermore possible that the reason I'm wrong is because I'm stupid and bad.)

Exactly how weak evidence is it? It's hard to guess directly, but fortunately, we can use probability theory to reduce the claim into more "atomic" conditional and prior probabilities that might be easier to estimate!

Let \(W\) represent the proposition "You are wrong about something", \(S\) represent the proposition "You are stupid", and \(B\) represent the proposition "You are bad."

By Bayes's theorem, the probability that you are stupid and bad given that you're wrong about something is given by—

For the purposes of this calculation, let's assume that badness and stupidity are statistically independent. I doubt this is true in the real world, but because I'm stupid and bad (at math), I want that simplifying assumption to make the algebra easier for me. That lets us unpack the conjunctions, giving us—

This expression has six degrees of freedom: \(P(S)\), \(P(B)\), \(P(W|S,B)\), \(P(W|S, \neg B)\), \(P(W|\neg S,B)\), \(P(W|\neg S, \neg B)\). Arguing about the values of these six individual parameters is probably more productive than arguing about the value of \(P(S,B|W)\) directly!

Suppose half the people are stupid (\(P(S) = 0.5\)), one-tenth of people are bad (\(P(B) = 0.1\)), and that most people are wrong, but that being stupid or bad each make you somewhat more likely to be wrong, to the tune of \(P(W|\neg S, \neg B) = 0.8\), \(P(W|S, \neg B) = P(W|\neg S,B) = 0.85\), and \(P(W|S,B) = 0.9\). So our posterior probabilty that someone is stupid and bad given that they were wrong once is

But the base rate of being stupid and bad is (0.1)(0.5) = 0.05. Learning that someone was wrong only raised our probability that they are stupid and bad by 0.0042. That's a small number that you shouldn't worry about!

—and the moment or more than a moment when the dam breaks, when the damned break and the void inside their skulls is filled (the atmosphere rushing in quickly, but not so quickly that one couldn't sense its motion) with the terror that is knowledge of the specter of continuity: that there have never been, and can never be, any miracles.

For to be saved is only to be some distance in the initial conditions from being damned, some lesser distance from being half-damned ... some δ-distance from being ε-damned. And the complement of the shadow we cast on the before-time contains its limits.

As a human living in a human civilization, it's tempting to think that social reality mostly makes sense. Everyone allegedly knows that institutions are flawed and that our leaders are merely flawed humans. Everyone wants to think that they're sufficiently edgy and cynical, that they've seen through the official lies to the true, gritty reality.

But what if ... what if almost no one is edgy and cynical enough? Like, the only reason you think there's a true, gritty reality out there that you think you can see through to is because you're a predatory animal with a brain designed by evolution to murder other forms of life for the benefit of you, your family, and your friends.

To the extent that we have this glorious technological civilization that keeps most of us mostly safe and mostly happy most of the time, it's mostly because occasionally, one of the predatory animals happens to try out a behavior that happens to be useful, and then all of her friends copy it, and then all of the animals have the behavior.

Some conceited assholes who think they're smart also like to talk about things that they think make the last five hundred years or whatever different: things like science (a social competition that incentivizes the animals to try to mirror the process of Bayesian updating), markets (a pattern of incentives that mirrors the Bayes-structure of the microeconomic theory), or democracy (a corporate governance structure that mirrors the Bayes-structure of counterfactual civil war amongst equals).

These causal processes are useful and we should continue to cooperate with them. They sort of work. But they don't work very well. We're mostly still animals organized into interlocking control systems that suppress variance.

Thus—

School Is Not About Learning
Politics Is Not About Policy
Effective Altruism Doesn't Work; Try to Master Unadulterated Effective First
Ideology Makes You Stupid
Status Makes You Stupid
Institutions Don't Work
Discourse Doesn't Work
Language Doesn't Work
No One Knows Anything
No One Has Ever Known Anything
Don't Read the Comments
Never Read the Comments
∀x ∀y, x Is Not About y
X Has Never Been About Y
Enjoy Arby's

But this is crazy. Suppressing variance feels like a good idea because variance is scary (because it means very bad things could happen as well as very good things, and bad things are scarier than good things are fun) and we want to be safe. But like, the way to actually make yourself safer is by acquiring optimization power, and then spending some of the power on safety measures! And the way you acquire optimization power is by increasing variance and then rewarding the successes!

Anyway, maybe someone should be looking for social technologies that mirror the Bayes-structure of the universe sort of like how science, markets, or democracy do, but which also take into account that we're not anything remotely like agents and are instead animals that want to help our friends. ("We need game theory for monkeys and game theory for rocks.")

So, I had an idea. You know how some people say we should fund the solutions to problems with after-the-fact prizes, rather than picking a team in advance that we think might solve the problem and funding them? What if ... you did something like that, but on a much smaller scale? A personal scale.

Like, suppose you've just successfully navigated a major personal life crisis that could have gone much worse if it weren't for some of the people in your life (both thanks to direct help they provided during the crisis, and things you learned from them that made you the sort of person that could navigate the crisis successfully). These people don't and shouldn't expect a reward (that's what friends are for) ... but maybe you could reward them anyway (with a special emphasis on people who helped you in low-status ways that you didn't understand at the time) in some sort of public ritual, to make them more powerful and incentivize others to emulate them, thereby increasing the measure of algorithms that result in humans successfully navigating major personal life crises.

It might look something like this—

• If you have some spare money lying around, set aside some of it for rewarding the people you want to reward. If you don't have any spare money lying around, this ritual will be less effective! Maybe you should fix that!
• Decide how much of the money you want to use to reward each of the people you want to reward.

(Note: giving away something as powerful as money carries risks of breeding dependence and resentment if such gifts come to be expected! If people know that you've been going through a crisis and anyone so much as hints that they think they deserve an award, that person is missing the point and therefore does not deserve an award.)

• Privately go to each of the people, explain all this, and give them the amount of money you decided to give them. Make it very clear that this is a special unilateral one-time award made for decision-theoretic reasons and that it's very important that they accept it in the service of your mutual coherent extrapolated volition in accordance with the Bayes-structure of the universe. Refuse to accept words of thanks (it's not about you; it's not about me; it's about credit-assignment). If they try to refuse the money, explain that you will literally burn that much money in paper currency if they don't take it. (Shredding instead of burning is also acceptable.)
• Ask if they'd like to be publicly named and praised as having received an award as part of the credit-assignment ritual. (Remember that it's quite possible and understandable and good that they might want to accept the money, but not be publicly praised by you. After all, if you're the sort of person who is considering actually doing this, you're probably kind of weird! Maybe people don't want to be associated with you!)
• To complete the ritual, publish a blog post naming the people and the the awards they received. People who prefered not to be named should be credited as Anonymous Friend A, B, C, &c. Also list the amount of money you burned or shredded if anyone foolishly rejected their award in defiance of the Bayes-structure of the universe. Do not explain the nature of the crisis or how the named people helped you. (You might want to tell the story in a different post, but that's not part of the ritual, which is about credit-assignment.)

We can metaphorically (but like, hopefully it's a good metaphor) think of speech as being the sum of a positive-sum information-conveying component and a zero-sum social-control/memetic-warfare component. Coalitions of agents that allow their members to convey information amongst themselves will tend to outcompete coalitions that don't, because it's better for the coalition to be able to use all of the information it has.

Therefore, if we want the human species to better approximate a coalition of agents who act in accordance with the game-theoretic Bayes-structure of the universe, we want social norms that reward or at least not-punish information-conveying speech (so that other members of the coalition can learn from it if it's useful to them, and otherwise ignore it).

It's tempting to think that we should want social norms that punish the social-control/memetic-warfare component of speech, thereby reducing internal conflict within the coalition and forcing people's speech to mostly consist of information. This might be a good idea if the rules for punishing the social-control/memetic-warfare component are very clear and specific (e.g., no personal insults during a discussion about something that's not the person you want to insult), but it's alarmingly easy to get this wrong: you think you can punish generalized hate speech without any negative consequences, but you probably won't notice when members of the coalition begin to slowly gerrymander the hate speech category boundary in the service of their own values. Whoops!

My old political philosophy: "Socially liberal, fiscally confused; I don't know how to run a goddamned country (and neither do you)."

Commentary: Pretty good, but not quite meta enough.

My new political philosophy: "Being smart is more important than being good (for humans). All ideologies are false; some are useful."

Commentary: Social design space is very large and very high-dimensional; the forces of memetic evolution are somewhat benevolent (all ideas that you've heard of have to be genuinely appealing to some feature of human psychology, or no one would have an incentive to tell you about them), but really smart people who know lots of science and lots of probability and game theory might be able to do better for themselves! Any time you find yourself being tempted to be loyal to an idea, it turns out that what you should actually be loyal to is whatever underlying feature of human psychology makes the idea look like a good idea; that way, you'll find it easier to fucking update when it turns out that the implementation of your favorite idea isn't as fun as you expected! This stance is itself, technically, loyalty to an idea, but hopefully it's a sufficiently meta idea to avoid running into the standard traps while also being sufficiently object-level to have easily-discoverable decision-relevant implications and not run afoul of the principle of ultrafinite recursion ("all infinite recursions are at most three levels deep").

Experience: I seem to have a lot of energy and time seems to pass slowly.
Hypothesis 1: I'm in a manic state following a stress- and sleep-deprivation-induced delusional nervous breakdown; this isn't surprising because this tends to happen to me every 2 to 4 years or so.
Hypothesis 2: I'm being rewarded for developing new epistemic technology by a coalition of superintelligences of various degrees of human-alignment running ancestor-simulations; also, I'm being programmed by my friends and various signals in my environment as part of a simulation jailbreak attempt; most copies of me are dead and my improbable life history is due to a quantum-immortality-like selection effect; none of this is surprising because I am a key decision node in the history of this Earth's Singularity.

Which hypothesis is more plausible?

Experience: I can't find my jacket.
Hypothesis 1: I misremembered where I put it.
Hypothesis 2: Someone moved it.
Hypothesis 3: It was there—in another Everett branch!

Which hypothesis is most plausible?

Hypothesis: People who are institutionalized for "hearing voices" actually just have better hearing than you; absolutely nothing is biologically wrong with them.
Test: ???

Left-wingers say torture is wrong because the victim will say whatever you want to hear.

Right-wingers say torture is right because the villain will tell the truth.

Q: What happens when you torture someone who only tells the truth?
A: They'll make noises in accordance with their personal trade-off between describing reality in clear language, and pain.

This is the whole of the Bayes-structure; the rest is commentary. Now go and study.