An Algorithmic Lucidity

(originally published at Less Wrong)

Adversarial Examples: A Problem

The apparent successes of the deep learning revolution conceal a dark underbelly. It may seem that we now know how to get computers to (say) check whether a photo is of a bird, but this façade of seemingly good performance is belied by the existence of adversarial examples—specially prepared data that looks ordinary to humans, but is seen radically differently by machine learning models.

The differentiable nature of neural networks, which make them possible to be trained at all, are also responsible for their downfall at the hands of an adversary. Deep learning models are fit using stochastic gradient descent (SGD) to approximate the function between expected inputs and outputs. Given an input, an expected output, and a loss function (which measures "how bad" it is for the actual output to differ from the expected output), we can calculate the gradient of the loss on the input—the derivative with respect to every parameter in our neural network—which tells us which direction to adjust the parameters in order to make the loss go down, to make the approximation better.¹

But gradients are a double-edged sword: the same properties that make it easy to calculate how to adjust a model to make it better at classifying an image, also make it easy to calculate how to adjust an image to make the model classify it incorrectly. If we take the gradient of the loss with respect to the pixels of the image (rather than the parameters of the model), that tells us which direction to adjust the pixels to make the loss go down—or up. (The direction of steepest increase is just the opposite of the direction of steepest decrease.) A tiny step in that direction in imagespace perturbs the pixels of an image just so—making this one the tiniest bit darker, that one the tiniest bit lighter—in a way that humans don't even notice, but which completely breaks an image classifier sensitive to that direction in the conjunction of many pixel-dimensions, making it report utmost confidence in nonsense classifications.

Some might ask: why does it matter if our image classifier fails on examples that have been mathematically constructed to fool it? If it works for the images one would naturally encounter, isn't that good enough?

One might mundanely reply that gracefully handling untrusted inputs is a desideratum for many real-world applications, but a more forward-thinking reply might instead emphasize what adversarial examples imply about our lack of understanding of the systems we're building, separately from whether we pragmatically expect to face an adversary. It's a problem if we think we've trained our machines to recognize birds, but they've actually learned to recognize a squiggly alien set in imagespace that includes a lot of obvious non-birds and excludes a lot of obvious birds. To plan good outcomes, we need to understand what's going on, and "The loss happens to increase in this direction" is at best only the start of a real explanation.

One obvious first guess as to what's going on is that the models are overfitting. Gradient descent isn't exactly a sophisticated algorithm. There's an intuition that the first solution that you happen to find by climbing down the loss landscape is likely to have idiosyncratic quirks on any inputs it wasn't trained for. (And that an AI designer from a more competent civilization would use a principled understanding of vision to come up with something much better than what we get by shoveling compute into SGD.) Similarly, a hastily cobbled-together conventional computer program that passed a test suite is going to have bugs in areas not covered by the tests.

But that explanation is in tension with other evidence, like the observation that adversarial examples often generalize between models. (An adversarial example optimized against one model is often misclassified by others, too, and even assigned the same class.) It seems unlikely that different hastily cobbled-together programs would have the same bug.

In "Adversarial Examples Are Not Bugs, They Are Features", Andrew Ilyas et al. propose an alternative explanation, that adversarial examples arise from predictively useful features that happen to not be robust to "pixel-level" perturbations. As far as the in-distribution predictive accuracy of the model is concerned, a high-frequency pattern that humans don't notice is fair game for distinguishing between image classes; there's no rule that the features that happen to be salient to humans need to take priority. Ilyas et al. provide some striking evidence for this thesis in the form of a model trained exclusively on adversarial examples yielding good performance on the original, unmodified test set (!!).² On this view, adversarial examples arise from gradient descent being "too smart", not "too dumb": the program is fine; if the test suite didn't imply the behavior we wanted, that's our problem.

On the other hand, there's also some evidence that gradient descent being "dumb" may play a role in adversarial examples, in conjunction with the counterintuitive properties of high-dimensional spaces. In "Adversarial Spheres", Justin Gilmer et al. investigated a simple synthetic dataset of two classes representing points on the surface of two concentric n-dimensional spheres of radiuses 1 and (an arbitrarily chosen) 1.3. For an architecture yielding an ellipsoidal decision boundary, training on a million datapoints produced a network with very high accuracy (no errors in 10 million samples), but for which most of the axes of the decision ellipsoid were wrong, lying inside the inner sphere or outside the outer sphere—implying the existence of on-distribution adversarial examples (points on one sphere classified by the network as belonging to the other). In high-dimensional space, pinning down the exact contours of the decision boundary is a bigger ask of SGD than merely being right virtually all of the time—even though a human wouldn't take a million datapoints to notice the hypothesis, "Hey, these all have a norm of exactly either 1 or 1.3."

Adversarial Training: A Solution?

Our story so far: we used gradient-based optimization to find a neural network that seemed to get low loss on an image classification task—that is, until an adversary used gradient-based optimization to find images on which our network gets high loss instead. Is that the end of the story? Are neural networks just the wrong idea for computer vision after all, or is there some way to continue within the current paradigm?

Would you believe that the solution involves ... gradient-based optimization?

In "Towards Deep Learning Models Resistant to Adversarial Attacks", Aleksander Madry et al. provide a formalization of the problem of adversarially robust classifiers. Instead of just trying to find network parameters \(\theta\) that minimize loss \(L\) on an input \(x\) of intended class \(y\), as in the original image classification task, the designers of a robust classifier are trying to minimize loss on inputs with a perturbation \(\delta\) crafted by an adversary trying to maximize loss (subject to some maximum perturbation size \(\varepsilon\)):

In this formulation, the attacker's problem of creating adversarial examples, and the defender's problem of training a model robust to them, are intimately related. If we change the image-classification problem statement to be about correctly classifying not just natural images, but an \(\varepsilon\)-ball around them, then you've defeated all adversarial examples up to that \(\varepsilon\). This turns out to generally require larger models than the classification problem for natural images: evidently, the decision boundary needed to separate famously "spiky" high-dimensional balls is significantly more complicated than that needed to separate natural inputs as points.

To solve the inner maximization problem, Madry et al. use the method of projected gradient descent (PGD) for constrained optimization: do SGD on the unconstrained problem, but after every step, project the result onto the constraint (in this case, the set of perturbations of size less than \(\varepsilon\)). This is somewhat more sophisticated than just generating any old adversarial examples and throwing them into your training set; the iterative aspect of PGD makes a difference.

Adversarial Robustness Is About Aligning Human and Model Decision Boundaries

What would it look like if we succeeded at training an adversarially robust classifier? How would you know if it worked? It's all well and good to say that a classifier is robust if there are no adversarial examples: you shouldn't be able to add barely-perceptible noise to an image and completely change the classification. But by the nature of the problem, adversarial examples aren't machine-checkable. We can't write a function that either finds them or reports "No solution found." The machine can only optimize for inputs that maximize loss. We, the humans, call such inputs "adversarial examples" when they look normal to us.

Imagespace is continuous: in the limit of large \(\varepsilon\), you can perturb any image into any other—just interpolate the pixels. When we say we want an adversarially robust classifier, we mean that perturbations that change the model's output should also make a human classify the input differently. Trying to find adversarial examples against a robust image classifier amounts to trying to find the smallest change to an image that alters what it "really" looks like (to humans).

You might wonder what the smallest such change could be, or perhaps if there even is any nontrivally "smallest" change (significantly better than just interpolating between images).

Madry et al. adversarially trained a classifier for the MNIST dataset of handwritten digits. Using PGD to search for adversarial examples under the \(\ell_2\) norm—the sum of the squares of the differences in pixel values between the original and perturbed images—the classifier's performance doesn't really tank until you crank \(\varepsilon\) up to around 4—at which point, the perturbations don't look like random noise anymore, as seen in Figure 12 from the paper:

Tasked with changing an image's class given a limited budget of how many pixels can be changed by how much, PGD concentrates its budget on human-meaningful changes—deleting part of the loop of a 9 to make a 7 or a 4, deleting the middle-left of an 8 to make a 3. In contrast to "vanilla" models whose susceptibility to adversarial examples makes us suspect their good performance on natural data is deceiving, it appears that the adversarially-trained model is seeing the same digits we are.

(I don't want to overstate the significance of this result and leave the impression that adversarial examples are necessarily "solved", but for the purposes of this post, I want to highlight the striking visual demonstration of what it looks like when adversarial training works.)³

An even more striking illustration of this phenomenon is provided in "Robustified ANNs Reveal Wormholes Between Human Category Percepts" by Guy Gaziv, Michael J. Lee, and James J. DiCarlo.⁴

The reason adversarial examples are surprising and disturbing is because they seem to reveal neural nets as fundamentally brittle in a way that humans aren't: we can't imagine our visual perception being so drastically effected by such small changes to an image. But what if that's just because we didn't know how to imagine the right changes?

Gaziv et al. adversarially trained image classifier models to be robust against perturbations under the \(\ell_2\) norm of \(\varepsilon\) being 1, 3, or 10, and then tried to produce adversarial examples with \(\epsilon\) up to 30.⁵ (For 224×224 images in the RGB colorspace, the maximum possible \(\ell_2\) distance is \(\sqrt{3 \cdot 224^2} \approx 388\). The typical difference between ImageNet images is about 130.)

What they found is that adversarial examples optimized to change the robustified models' classifications also changed human judgments, as confirmed in experiments where subjects were shown the images for up to 0.8 seconds—but you can also see for yourself in the paper or on the project website. Here's Figure 3a from the paper:

The authors confirm in the Supplementary Material that random \(\epsilon\) = 30 perturbations don't affect human judgments at all. (Try squinting or standing far away from the monitor to better appreciate just how similar the pictures in Figure 3a are.) The robustified models are close enough to seeing the same animals we are that adversarial attacks against them are also attacks against us, precisely targeting their limited pixel-changing budget on surprising low-\(\ell_2\)-norm "wormholes" between apparently distant human precepts.

Implications for Alignment?

Futurists have sometimes worried that our civilization's coming transition to machine intelligence may prove to be incompatible with human existence. If AI doesn't see the world the same way as we do, then there's no reason for it to steer towards world-states that we would regard as valuable. (Having a concept of the right thing is a necessary if not sufficient prerequisite for doing the right thing.)

As primitive precursors to machine intelligence have been invented, some authors have taken the capabilities of neural networks to learn complicated functions as an encouraging sign. Early discussions of AI alignment had emphasized that "leaving out just [...] one thing" could result in a catastrophic outcome—for example, a powerful agent that valued subjective experience but lacked an analogue of boredom would presumably use all its resources to tile the universe with repetitions of its most optimized experience. (The emotion of boredom is evolution's solution to the exploration–exploitation trade-off; there's no reason to implement it if you can just compute the optimal policy.)

The particular failure mode of "leaving one thing out" is starting to seem less likely on the current paradigm. Katja Grace notes that image synthesis methods have no trouble generating photorealistic human faces. Diffusion models don't "accidentally forget" that faces have nostrils, even if a human programmer trying to manually write a face image generation routine might. Similarly, large language models obey the quantity-opinion-size-age-shape-color-origin-purpose adjective order convention in English without the system designers needing to explicitly program that in or even be aware of it, despite the intuitive appeal of philosophical arguments one could make to the effect that "English is fragile." So the optimistic argument goes: if instilling human values into future AGI is as easy as specifying desired behavior for contemporary generative AI, then we might be in luck?

But even if machine learning methods make some kinds of failures due to brittle specification less likely, that doesn't imply that alignment is easy. A different way things could go wrong is if representations learned from data turn out not to be robust off the training distribution. A function that tells your AI system whether an action looks good and is right virtually all of the time on natural inputs isn't safe if you use it to drive an enormous search for unnatural (highly optimized) inputs on which it might behave very differently.

Thus, the extent to which ML methods can be made robust is potentially a key crux for views about the future of Earth-originating intelligent life. In a 2018 comment on a summary of Paul Christiano's research agenda, Eliezer Yudkowsky characterized one of his "two critical points" of disagreement with Christiano as being about how easy robust ML is:

Eliezer expects great Project Chaos and Software Despair from trying to use gradient descent, genetic algorithms, or anything like that, as the basic optimization to reproduce par-human cognition within a boundary in great fidelity to that boundary as the boundary was implied by human-labeled data. Eliezer thinks that if you have any optimization powerful enough to reproduce humanlike cognition inside a detailed boundary by looking at a human-labeled dataset trying to outline the boundary, the thing doing the optimization is powerful enough that we cannot assume its neutrality the way we can assume the neutrality of gradient descent.

Eliezer expects weird squiggles from gradient descent—it's not that gradient descent can never produce par-human cognition, even natural selection will do that if you dump in enough computing power. But you will get the kind of weird squiggles in the learned function that adversarial examples expose in current nets—special inputs that weren't in the training distribution, but look like typical members of the training distribution from the perspective of the training distribution itself, will break what we think is the intended labeling from outside the system. Eliezer does not think Ian Goodfellow will have created a competitive form of supervised learning by gradient descent which lacks "squiggles" findable by powerful intelligence by the time anyone is trying to create ML-based AGI, though Eliezer is certainly cheering Goodfellow on about this and would recommend allocating Goodfellow $1 billion if Goodfellow said he could productively use it. You cannot iron out the squiggles just by using more computing power in bounded in-universe amounts.

Christiano replied, in part:

For adversarial examples in particular, I think that the most reasonable guess right now is that it takes more model capacity (and hence data) to classify all perturbations of natural images correctly rather than merely classifying most correctly—i.e., the smallest neural net that classifies them all right is bigger than the smallest neural net that gets most of them right—but that if you had enough capacity+data then adversarial training would probably be robust to adversarial perturbations. Do you want to make the opposite prediction?

At the time in 2018, it may have been hard for readers to determine which of these views was less wrong—and maybe it's still too early to call. ("Robust ML" is an active research area, not a crisp problem statement that we can definitively say is solved or not-solved.) But it should be a relatively easier call for the ArXiv followers of 2024 than the blog readers of 2018, as the state of the art has advanced and more relevant experiments have been published. To my inexpert eyes, the Gaziv et al. "perceptual wormholes" result does seem like a clue that "ironing out the squiggles" may prove to be feasible after all—that adversarial examples are mostly explainable in terms of non-robust features and high-dimensional geometry, and remediable by better (perhaps more compute-intensive) methods—rather than being a fundamental indictment of our Society's entire paradigm for building AI.

Am I missing anything important? Probably. I can only hope that someone who isn't will let me know in the comments.

(originally published at Less Wrong)

(Epistemic status: publication date is significant.)

Some observers have argued that the totality of "AI safety" and "alignment" efforts to date have plausibly had a negative rather than positive impact on the ultimate prospects for safe and aligned artificial general intelligence. This perverse outcome is possible because research "intended" to help with AI alignment can have a larger impact on AI capabilities, moving existentially-risky systems closer to us in time without making corresponding cumulative progress on the alignment problem.

When things are going poorly, one is often inclined to ask "when it all went wrong." In this context, some identify the founding of OpenAI in 2015 as a turning point, being casually downstream of safety concerns despite the fact no one who had been thinking seriously about existential risk thought the original vision of OpenAI was a good idea.

But if we're thinking about counterfactual impacts on outcomes, rather than grading the performance of the contemporary existential-risk-reduction movement in particular, it makes sense to posit earlier turning points.

Perhaps—much earlier. Foresighted thinkers such as Marvin Minsky (1960), Alan Turing (1951), and George Eliot (1879!!) had pointed to AI takeover as something that would likely happen eventually—is the failure theirs for not starting preparations earlier? Should we go back even earlier, and blame the ancient Greeks for failing to discover evolution and therefore adopt a eugenics program that would have given their descendants higher biological intelligence with which to solve the machine intelligence alignment problem?

Or—even earlier? There's an idea that humans are the stupidest possible creatures that could have built a technological civilization: if it could have happened at a lower level of intelligence, it would have (and higher intelligence would have no time to evolve).

But intelligence isn't the only input into our species's penchant for technology; our hands with opposable thumbs are well-suited for making and using tools, even though the proto-hands of our ancestors were directly adapted for climbing trees. An equally-intelligent species with a less "lucky" body plan or habitat, similar to crows (lacking hands) or octopuses (living underwater, where, e.g., fires cannot start), might not have gotten started down the path of cultural accumulation of technology—even while a more intelligent crow- or octopus-analogue might have done so.

It's plausible that the values of humans and biological aliens overlap to a much higher degree than those of humans and AIs; we should be "happy for" other biological species that solve their alignment problem, even if their technologically-mature utopia is different from the one we would create.

But that being the case, it follows that we should regard some alien civilizations as more valuable than our own, whenever the difference in values is outweighed by a sufficiently large increase in the probability of solving the alignment problem. (Most of the value of ancestral civilizations lies in the machine superintelligences that they set off, because ancestral civilizations are small and the Future is big.) If opposable thumbs were more differentially favorable to AI capabilities than AI alignment, we should perhaps regard the evolution of humans as a tragedy: we should prefer to go extinct and be replaced by some other species that needed a higher level of intelligence in order to wield technology. The evolution of humans was net-negative for human values.

A Surprising Development in the Study of Multi-layer Parameterized Graphical Function Approximators

As a programmer and epistemology enthusiast, I've been studying some statistical modeling techniques lately! It's been boodles of fun, and might even prove useful in a future dayjob if I decide to pivot my career away from the backend web development roles I've taken in the past.

More specifically, I've mostly been focused on multi-layer parameterized graphical function approximators, which map inputs to outputs via a sequence of affine transformations composed with nonlinear "activation" functions.

(Some authors call these "deep neural networks" for some reason, but I like my name better.)

It's a curve-fitting technique: by setting the multiplicative factors and additive terms appropriately, multi-layer parameterized graphical function approximators can approximate any function. For a popular choice of "activation" rule which takes the maximum of the input and zero, the curve is specifically a piecewise-linear function. We iteratively improve the approximation f(x, θ) by adjusting the parameters θ in the direction of the derivative of some error metric on the current approximation's fit to some example input–output pairs (x, y), which some authors call "gradient descent" for some reason. (The mean squared error (f(x, θ) − y)² is a popular choice for the error metric, as is the negative log likelihood −log P(y | f(x, θ)). Some authors call these "loss functions" for some reason.)

Basically, the big empirical surprise of the previous decade is that given a lot of desired input–output pairs (x, y) and the proper engineering know-how, you can use large amounts of computing power to find parameters θ to fit a function approximator that "generalizes" well—meaning that if you compute ŷ = f(x, θ) for some x that wasn't in any of your original example input–output pairs (which some authors call "training" data for some reason), it turns out that ŷ is usually pretty similar to the y you would have used in an example (x, y) pair.

It wasn't obvious beforehand that this would work! You'd expect that if your function approximator has more parameters than you have example input–output pairs, it would overfit, implementing a complicated function that reproduced the example input–output pairs but outputted crazy nonsense for other choices of x—the more expressive function approximator proving useless for the lack of evidence to pin down the correct approximation.

And that is what we see for function approximators with only slightly more parameters than example input–output pairs, but for sufficiently large function approximators, the trend reverses and "generalization" improves—the more expressive function approximator proving useful after all, as it admits algorithmically simpler functions that fit the example pairs.

The other week I was talking about this to an acquaintance who seemed puzzled by my explanation. "What are the preconditions for this intuition about neural networks as function approximators?" they asked. (I paraphrase only slightly.) "I would assume this is true under specific conditions," they continued, "but I don't think we should expect such niceness to hold under capability increases. Why should we expect this to carry forward?"

I don't know where this person was getting their information, but this made zero sense to me. I mean, okay, when you increase the number of parameters in your function approximator, it gets better at representing more complicated functions, which I guess you could describe as "capability increases"?

But multi-layer parameterized graphical function approximators created by iteratively using the derivative of some error metric to improve the quality of the approximation are still, actually, function approximators. Piecewise-linear functions are still piecewise-linear functions even when there are a lot of pieces. What did you think it was doing?

Multi-layer Parameterized Graphical Function Approximators Have Many Exciting Applications

To be clear, you can do a lot with function approximation!

For example, if you assemble a collection of desired input–output pairs (x, y) where the x is an array of pixels depicting a handwritten digit and y is a character representing which digit, then you can fit a "convolutional" multi-layer parameterized graphical function approximator to approximate the function from pixel-arrays to digits—effectively allowing computers to read handwriting.

Such techniques have proven useful in all sorts of domains where a task can be conceptualized as a function from one data distribution to another: image synthesis, voice recognition, recommender systems—you name it. Famously, by approximating the next-token function in tokenized internet text, large language models can answer questions, write code, and perform other natural-language understanding tasks.

I could see how someone reading about computer systems performing cognitive tasks previously thought to require intelligence might be alarmed—and become further alarmed when reading that these systems are "trained" rather than coded in the manner of traditional computer programs. The summary evokes imagery of training a wild animal that might turn on us the moment it can seize power and reward itself rather than being dependent on its masters.

But "training" is just a suggestive name. It's true that we don't have a mechanistic understanding of how function approximators perform tasks, in contrast to traditional computer programs whose source code was written by a human. It's plausible that this opacity represents grave risks, if we create powerful systems that we don't know how to debug.

But whatever the real risks are, any hope of mitigating them is going to depend on acquiring the most accurate possible understanding of the problem. If the problem is itself largely one of our own lack of understanding, it helps to be specific about exactly which parts we do and don't understand, rather than surrendering the entire field to a blurry aura of mystery and despair.

An Example of Applying Multi-layer Parameterized Graphical Function Approximators in Success-Antecedent Computation Boosting

One of the exciting things about multi-layer parameterized graphical function approximators is that they can be combined with other methods for the automation of cognitive tasks (which is usually called "computing", but some authors say "artificial intelligence" for some reason).

In the spirit of being specific about exactly which parts we do and don't understand, I want to talk about Mnih et al. 2013's work on getting computers to play classic Atari games (like Pong, Breakout, or Space Invaders). This work is notable as one of the first high-profile examples of using multi-layer parameterized graphical function approximators in conjunction with success-antecedent computation boosting (which some authors call "reinforcement learning" for some reason).

If you only read the news—if you're not in tune with there being things to read besides news—I could see this result being quite alarming. Digital brains learning to play video games at superhuman levels from the raw pixels, rather than because a programmer sat down to write an automation policy for that particular game? Are we not already in the shadow of the coming race?

But people who read textbooks and not just news, being no less impressed by the result, are often inclined to take a subtler lesson from any particular headline-grabbing advance.

Mnih et al.'s Atari result built off the technique of Q-learning introduced two decades prior. Given a discrete-time present-state-based outcome-valued stochastic control problem (which some authors call a "Markov decision process" for some reason), Q-learning concerns itself with defining a function Q(s, a) that describes the value of taking action a while in state s, for some discrete sets of states and actions. For example, to describe the problem faced by an policy for a grid-based video game, the states might be the squares of the grid, and the available actions might be moving left, right, up, or down. The Q-value for being on a particular square and taking the move-right action might be the expected change in the game's score from doing that (including a scaled-down expectation of score changes from future actions after that).

Upon finding itself in a particular state s, a Q-learning policy will usually perform the action with the highest Q(s, a), "exploiting" its current beliefs about the environment, but with some probability it will "explore" by taking a random action. The predicted outcomes of its decisions are compared to the actual outcomes to update the function Q(s, a), which can simply be represented as a table with as many rows as there are possible states and as many columns as there are possible actions. We have theorems to the effect that as the policy thoroughly explores the environment, it will eventually converge on the correct Q(s, a).

But Q-learning as originally conceived doesn't work for the Atari games studied by Mnih et al., because it assumes a discrete set of possible states that could be represented with the rows in a table. This is intractable for problems where the state of the environment varies continuously. If a "state" in Pong is a 6-tuple of floating-point numbers representing the player's paddle position, the opponent's paddle position, and the x- and y-coordinates of the ball's position and velocity, then there's no way for the traditional Q-learning algorithm to base its behavior on its past experiences without having already seen that exact conjunction of paddle positions, ball position, and ball velocity, which almost never happens. So Mnih et al.'s great innovation was—

(Wait for it ...)

—to replace the table representing Q(s, a) with a multi-layer parameterized graphical function approximator! By approximating the mapping from state–action pairs to discounted-sums-of-"rewards", the "neural network" allows the policy to "generalize" from its experience, taking similar actions in relevantly similar states, without having visited those exact states before. There are a few other minor technical details needed to make it work well, but that's the big idea.

And understanding the big idea probably changes your perspective on the headline-grabbing advance. (It certainly did for me.) "Deep learning is like evolving brains; it solves problems and we don't know how" is an importantly different story from "We swapped out a table for a multi-layer parameterized graphical function approximator in this specific success-antecedent computation boosting algorithm, and now it can handle continuous state spaces."

Risks From Learned Approximation

When I solicited reading recommendations from people who ought to know about risks of harm from statistical modeling techniques, I was directed to a list of reputedly fatal-to-humanity problems, or "lethalities".

Unfortunately, I don't think I'm qualified to evaluate the list as a whole; I would seem to lack some necessary context. (The author keeps using the term "AGI" without defining it, and adjusted gross income doesn't make sense in context.)

What I can say is that when the list discusses the kinds of statistical modeling techniques I've been studying lately, it starts to talk funny. I don't think someone who's been reading the same textbooks as I have (like Prince 2023 or Bishop and Bishop 2024) would write like this:

Even if you train really hard on an exact loss function, that doesn't thereby create an explicit internal representation of the loss function inside an AI that then continues to pursue that exact loss function in distribution-shifted environments. Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction. [...] This is sufficient on its own [...] to trash entire categories of naive alignment proposals which assume that if you optimize a bunch on a loss function calculated using some simple concept, you get perfect inner alignment on that concept.

To be clear, I agree that if you fit a function approximator by iteratively adjusting its parameters in the direction of the derivative of some loss function on example input–output pairs, that doesn't create an explicit internal representation of the loss function inside the function approximator.

It's just—why would you want that? And really, what would that even mean? If I use the mean squared error loss function to approximate a set of data points in the plane with a line (which some authors call a "linear regression model" for some reason), obviously the line itself does not somehow contain a representation of general squared-error-minimization. The line is just a line. The loss function defines how my choice of line responds to the data I'm trying to approximate with the line. (The mean squared error has some elegant mathematical properties, but is more sensitive to outliers than the mean absolute error.)

It's the same thing for piecewise-linear functions defined by multi-layer parameterized graphical function approximators: the model is the dataset. It's just not meaningful to talk about what a loss function implies, independently of the training data. (Mean squared error of what? Negative log likelihood of what? Finish the sentence!)

This confusion about loss functions seems to be linked to a particular theory of how statistical modeling techniques might be dangerous, in which "outer" training results in the emergence of an "inner" intelligent agent. If you expect that, and you expect intelligent agents to have a "utility function", you might be inclined to think of "gradient descent" "training" as trying to transfer an outer "loss function" into an inner "utility function", and perhaps to think that the attempted transfer primarily doesn't work because "gradient descent" is an insufficiently powerful optimization method.

I guess the emergence of inner agents might be possible? I can't rule it out. ("Functions" are very general, so I can't claim that a function approximator could never implement an agent.) Maybe it would happen at some scale?

But taking the technology in front of us at face value, that's not my default guess at how the machine intelligence transition would go down. If I had to guess, I'd imagine someone deliberately building an agent using function approximators as a critical component, rather than your function approximator secretly having an agent inside of it.

That's a different threat model! If you're trying to build a good agent, or trying to prohibit people from building bad agents using coordinated violence (which some authors call "regulation" for some reason), it matters what your threat model is!

(Statistical modeling engineer Jack Gallagher has described his experience of this debate as "like trying to discuss crash test methodology with people who insist that the wheels must be made of little cars, because how else would they move forward like a car does?")

I don't know how to build a general agent, but contemporary computing research offers clues as to how function approximators can be composed with other components to build systems that perform cognitive tasks.

Consider AlphaGo and its successor AlphaZero. In AlphaGo, one function approximator is used to approximate a function from board states to move probabilities. Another is used to approximate the function from board states to game outcomes, where the outcome is +1 when one player has certainly won, −1 when the other player has certainly won, and a proportionately intermediate value indicating who has the advantage when the outcome is still uncertain. The system plays both sides of a game, using the board-state-to-move-probability function and board-state-to-game-outcome function as heuristics to guide a search algorithm which some authors call "Monte Carlo tree search". The board-state-to-move-probability function approximation is improved by adjusting its parameters in the direction of the derivative of its cross-entropy with the move distribution found by the search algorithm. The board-state-to-game-outcome function approximation is improved by adjusting its parameters in the direction of the derivative of its squared difference with the self-play game's ultimate outcome.

This kind of design is not trivially safe. A similarly superhuman system that operated in the real world (instead of the restricted world of board games) that iteratively improved an action-to-money-in-this-bank-account function seems like it would have undesirable consequences, because if the search discovered that theft or fraud increased the amount of money in the bank account, then the action-to-money function approximator would generalizably steer the system into doing more theft and fraud.

Statistical modeling engineers have a saying: if you're surprised by what your nerual net is doing, you haven't looked at your training data closely enough. The problem in this hypothetical scenario is not that multi-layer parameterized graphical function approximators are inherently unpredictable, or must necessarily contain a power-seeking consequentialist agent in order to do any useful cognitive work. The problem is that you're approximating the wrong function and get what you measure. The failure would still occur if the function approximator "generalizes" from its "training" data the way you'd expect. (If you can recognize fraud and theft, it's easy enough to just not use that data as examples to approximate, but by hypothesis, this system is only looking at the account balance.) This doesn't itself rule out more careful designs that use function approximators to approximate known-trustworthy processes and don't search harder than their representation of value can support.

This may be cold comfort to people who anticipate a competitive future in which cognitive automation designs that more carefully respect human values will foreseeably fail to keep up with the frontier of more powerful systems that do search harder. It may not matter to the long-run future of the universe that you can build helpful and harmless language agents today, if your civilization gets eaten by more powerful and unfriendlier cognitive automation designs some number of years down the line. As a humble programmer and epistemology enthusiast, I have no assurances to offer, no principle or theory to guarantee everything will turn out all right in the end. Just a conviction that, whatever challenges confront us in the future, we'll be a better position to face them by understanding the problem in as much detail as possible.

Bibliography

Bardo, Richard S., and Andrew G. Sutton. 2024. Reinforcement Learning. 2nd ed. Cambridge, MA: MIT Press.

Bishop, Christopher M., and Andrew M. Bishop. 2024. Deep Learning: Foundations and Concepts. Cambridge, UK: Cambridge University Press. https://www.bishopbook.com/

Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. "Playing Atari with Deep Reinforcement Learning." https://arxiv.org/abs/1312.5602

Prince, Simon J.D. 2023. Understanding Deep Learning. Cambridge, MA: MIT Press. http://udlbook.com/

(originally published at Less Wrong)

[Setting: a suburban house. The interior of the house takes up most of the stage; on the audience's right, we see a wall in cross-section, and a front porch. Simplicia enters stage left and rings the doorbell.]

Doomimir: [opening the door] Well? What do you want?

Simplicia: I can't stop thinking about our last conversation. It was kind of all over the place. If you're willing, I'd like to continue, but focusing in narrower detail on a couple points I'm still confused about.

Doomimir: And why should I bother tutoring an Earthling in alignment theory? If you didn't get it from the empty string, and you didn't get it from our last discussion, why should I have any hope of you learning this time? And even if you did, what good would it do?

Simplicia: [serenely] If the world is ending either way, I think it's more dignified that I understand exactly why. [A beat.] Sorry, that doesn't explain what's in it for you. That's why I had to ask.

Doomimir: [grimly] As you say. If this world is ending either way.

[He motions for her to come in, and they sit down.]

Doomimir: What are you confused about? I mean, that you wanted to talk about.

Simplicia: You seemed really intent on a particular intuition pump against human-imitation-based alignment strategies, where you compared LLMs to an alien actress. I didn't find that compelling.

Doomimir: But you claim to understand that LLMs that emit plausibly human-written text aren't human. Thus, the AI is not the character it's playing. Similarly, being able to predict the conversation in a bar, doesn't make you drunk. What's there not to get, even for you?

Simplicia: Why doesn't the "predicting barroom conversation doesn't make you drunk" analogy falsely imply "predicting the answers to modular arithmetic problems doesn't mean you implement modular arithmetic"?

Doomimir: To predict the conversation in a bar, you need to know everything the drunk people know, separately and in addition to everything you know. Being drunk yourself would just get in the way. Similarly, predicting the behavior of nice people isn't the same thing as being nice. Modular arithmetic isn't like that; there's nothing besides the knowledge to not implement.

Simplicia: But we only need our AI to compute nice behavior, not necessarily to have some internal structure corresponding to the quale of niceness. As far as safety properties go, we don't care whether the actress is "really drunk" as long as she stays in character.

Doomimir: [scoffing] Have you tried imagining any internal mechanisms at all other than a bare, featureless inclination to emit the outward behavior you observe?

Simplicia: [unfazed] Sure, let's talk about internal mechanisms. The reason I chose modular arithmetic as an example is because it's a task for which we have good interpretability results. Train a shallow transformer on a subset of the addition problems modulo some fixed prime. The network learns to map the inputs onto a circle in the embedding space, and then does some trigonometry to extract the residue, much as one would count forward on the face of an analog clock.

Alternatively, with a slightly different architecture that has a harder time with trig, it can learn a different algorithm: the embeddings are still on a circle, but the answer is computed by looking at the average of the embedding vectors of the inputs. On the face of an analog clock, the internal midpoints between distinct numbers that sum to 6 mod 12—that's 2 and 4, or 1 and 5, or 6 and 12, or 10 and 8, or 11 and 7—all lie on the line connecting 3 and 9. Thus, the sum-mod-p of two numbers can be determined by which line the midpoint of the inputs falls on—as long as the inputs aren't on opposite sides of the circle, in which case their midpoint is in the center, where all the lines meet. But the network compensates for such antipodal points by also learning another circle in a different subspace of the embedding space, such that inputs that are antipodal on the first circle are close together on the second, which helps disambiguate the answer.

Doomimir: Cute results. Excellent work—by Earth standards. And entirely unsurprising. Sure, if you train your neural net on a well-posed mathematical problem with a consistent solution, it will converge on a solution to that problem. What's your point?

Simplicia: It's evidence about the feasibility of learning desired behavior from training data. You seem to think that it's hopelessly naïve to imagine that training on "nice" data could result in generalizably nice behavior—that the only reason someone might think that was a viable path was is if they were engaging in magical reasoning about surface similarities. I think it's germane to point out that at least for this toy problem, we have a pretty concrete, non-magical story about how optimizing against a training set discovers an algorithm that reproduces the training data and also generalizes correctly to the test set.

For non-toy problems, we know empirically that deep learning can hit very precise behavioral targets: the vast hypermajority of programs don't speak fluent English or generate beautiful photorealistic images, and yet GPT-4 and Midjourney exist.

If doing that for "text" and "images" was a mere engineering problem, I don't see what fundamental theoretical barrier rules out the possibility of pulling off the same kind of thing for "friendly and moral real-world decisionmaking"—learning a "good person" or "obedient servant" function from data, much as Midjourney has learned a "good art" function.

It's true that diffusion models don't work like a human artist on the inside, but it's not clear why that matters? It would seem idle to retort, "Predicting what good art would look like, doesn't make you a good artist; having an æsthetic sense yourself would just get in the way", when you can actually use it to do a commissioned artist's job.

Doomimir: Messier tasks aren't going to have a unique solution like modular arithmetic. If genetic algorithms, gradient descent, or anything like that happens to hill-climb its way into something that appears to work, the function it learns is going to have all sorts of weird squiggles around inputs that we would call adversarial examples, that look like typical members of the training distribution from the AI's perspective, but not ours—which kill you when optimized over by a powerful AGI.

Simplicia: It sounds like you're making an empirical claim that solutions found by black-box optimization are necessarily contingent and brittle, but there's some striking evidence that seemingly "messy" tasks admit much more convergent solutions than one might expect. For example, on the surface, the word2vec and FastText word embeddings look completely different—as befitting being produced by two different codebases trained on different datasets. But when you convert their latent spaces to a relative representation—choosing some shared vocabulary words as anchors, and defining all other word vectors by their cosine similarities to the anchors—they look extremely similar.

It would seem that "English word embeddings" are a well-posed mathematical problem with a consistent solution. The statistical signature of the language as it is spoken is enough to pin down the essential structure of the embedding.

Relatedly, you bring up adversarial examples in a way that suggests that you think of them as defects of a primitive optimization paradigm, but it turns out that adversarial examples often correspond to predictively useful features that the network is actively using for classification, despite those features not being robust to pixel-level perturbations that humans don't notice—which I guess you could characterize as "weird squiggles" from our perspective, but the etiology of the squiggles presents a much more optimistic story about fixing the problem with adversarial training than if you thought "squiggles" were an inevitable consequence of using conventional ML techniques.

Doomimir: This is all very interesting, but I don't think it bears much on the reasons we're all going to die. It's all still on the "is" side of the is–ought gap. What makes intelligence useful—and dangerous—isn't a fixed repertoire of behaviors. It's search, optimization—the systematic discovery of new behaviors to achieve goals despite a changing environment. I don't think recent capabilities advances bear on the shape of the alignment challenge because being able to learn complex behavior on the training distribution was never what the problem was about.

Indeed, as long as we continue to be stuck in the paradigm of reasoning about "the training distribution"—growing minds rather than designing them—then we're not learning anything about how to aim cognition at specific targets—certainly not in a way that will hold up to dumping large amounts of optimization power into the system. The lack of an explicit "goal slot" in your neural network doesn't mean it's not doing any dangerous optimization; it just means you don't know what it is.

Simplicia: I think we can form educated guesses—

Doomimir: [interrupting] Guesses!

Simplicia: —probabilistic beliefs—about what kinds of optimization is being done by a system and whether it's a problem, even without a complete mechanistic interpretability story. If you think LLMs or future variations thereof are unsafe because they're analogous to an actress with her own goals playing a drunk character without herself being drunk, shouldn't that make some sort of testable prediction about their generalization behavior?

Doomimir: Nonfatally testable? Not necessarily. If you lend a con man $5, and he gives it back, that doesn't mean that you can trust him with larger amounts of money, if he only gave back the $5 because he hoped you would trust him with more.

Simplicia: Okay, I agree that deceptive alignment is potentially a real problem at some point, but can we at least distinguish between misgeneralization and deceptive alignment?

Doomimir: Mis-generalization? The goals you wanted aren't a property of the training data itself. The danger comes from correct generalization implying something you don't want.

Simplicia: Can I call it mal-generalization?

Doomimir: Sure.

Simplicia: So there are obviously risks from malgeneralization, where the network that fits your training distribution turns out to not behave the way you wanted against a different distribution. For example, a reinforcement learning policy trained to collect a coin at the right edge of a video game level, might end up continuing to navigate to the right edge of levels where the coin is in a different location. That's a worrying clue that if we misunderstand how inductive biases work and aren't careful with our training setup, we might train the wrong thing. As our civilization delegates more and more cognitive labor to machines, eventually humans will lose the ability to course-correct. We're starting to see the early signs of this: as I mentioned the other day, Anthropic Claude's preachy, condescending personality already gives me the creeps. I'm pretty nervous about extrapolating that into a future where all productive roles in Society are filled by Claude's children, concurrently with a transition to explosive economic growth rates.

But the malgeneralization examples I named aren't surprising when you look at how the systems were trained. For the game policy, "going to the coin" and "going to the right" did amount to the same thing in training—and randomizing the coin position in just a couple percent of training episodes suffices to instill the correct behavior. Regarding Claude, Anthropic is using a reinforcement-learning-from-AI-feedback method they call Constitutional AI: instead of having humans provide the labels for RLHF, they write up a list of principles, and have another language model do the labeling. It makes sense that a language model agent trained to conform to principles chosen by a committee at a California public benefit corporation would act like that.

In contrast, when you make analogies about an actress playing a drunk character not being drunk, or giving a con man $5, it doesn't sound like you're talking about the risk of training the wrong thing, where it's usually clear in retrospect if not foresight how training encouraged the bad behavior. Rather, it sounds like you don't think training can shape motivations—"inner" motivations—at all.

You might be talking about deceptive alignment, a hypothesized phenomenon where a situationally aware AI strategically feigns aligned behavior in order to preserve its later influence. Researchers have debated how likely that is, but I'm not sure what to make of those arguments. I'd like to factor that consideration out. Suppose, arguendo, that we could figure out how to avoid deceptive alignment. How would your risk story change?

Doomimir: What would that even mean? What we would think of as "deception" isn't a weird edge case you can trivially avoid; it's convergent for any agent that isn't specifically coordinating with you to interpret certain states of reality as communication signals with a shared meaning.

When you set out poisoned ant baits, you likely don't think of yourself as trying to deceive the ants, but you are. Similarly, a smart AI won't think of itself as trying to deceive us. It's trying to achieve its goals. If its plans happen to involve emitting sound waves or character sequences that we interpret as claims about the world, that's our problem.

Simplicia: "What would that even"—this isn't 2008, Doomishko! I'm talking about the technology right here in front of us! When GPT-4 writes original code for me, I don't think it's strategically deciding that obeying me instrumentally serves its final goals! From everything I've read about how it was made and seen about how it behaves, it looks awfully like it's just generalizing from its training distribution in an intuitively reasonable way. You ridicule people who deride LLMs as stochastic parrots, ignoring the obvious sparks of AGI right in front of their face. Why is it not equally absurd to deny the evidence in front of your face that alignment may be somewhat easier than it looked 15 years ago? By all means, expound on the nonobvious game theory of deception; by all means, point out that the superintelligence at the end of time will be an expected utility maximizer. But all the same, RLHF/DPO as the cherry on top of a cake of unsupervised learning is verifiably working miracles for us today—in response to commands, not because it has a will of its own aligned with ours. Why is that merely "capabilities" and not at all "alignment"? I'm trying to understand, Doomimir Doomovitch, but you're not making this easy!

Doomimir: [starting to anger] Simplicia Optimistovna, if you weren't from Earth, I'd say I don't think you're trying to understand. I never claimed that GPT-4 in particular is what you would call deceptively aligned. Endpoints are easier to predict than intermediate trajectories. I'm talking about what will happen inside almost any sufficiently powerful AGI, by virtue of it being sufficiently powerful.

Simplicia: But if you're only talking about the superintelligence at the end of time—

Doomimir: [interrupting] This happens significantly before that.

Simplicia: —and not making any claims about existing systems, then what was the whole "alien actress", "predicting bar conversations doesn't make you drunk" analogy about? If it was just a ham-fisted way to explain to normies that LLMs that do relatively well on a Turing test aren't humans, then I agree, trivially. But it seemed like you thought you were making a much stronger point, ruling out an entire class of alignment strategies based on imitation.

Doomimir: [cooler] Basically, I think you're systematically failing to appreciate how things that have been optimized to look good to you can predictably behave differently in domains where they haven't been optimized to look good to you—particularly, when they're doing any serious optimization of their own. You mention the video game agent that navigates to the right instead of collecting a coin. You claim that it's not surprising given the training set-up, and can be fixed by appropriately diversifying the training data. But could you have called the specific failure in advance, rather than in retrospect? When you enter the regime of transformatively powerful systems, you do have to call it in advance.

I think if you understood what was really going on inside of LLMs, you'd see thousands and thousands of analogues of the "going right rather than getting the coin" problem. The point of the actress analogy is that the outward appearance doesn't tell you what goals the system is steering towards, which is where the promise and peril of AGI lies—and the fact that deep learning systems are a inscrutable mess, not all of which can be described as "steering towards goals", makes the situation worse, not better. The analogy doesn't depend on existing LLMs having the intelligence or situational awareness for the deadly failure modes to have already appeared, and it doesn't preclude LLMs being mundanely useful in the manner of an interactive textbook—much as an actress could be employed to give plausible-sounding answers to questions posed to her character, without being that character.

Simplicia: Those mismatches still need to show up in behavior under some conditions, though. I complained about Claude's personality, but that honestly seems fixable with scaling by an AI company not based in California. If human imitation is so superficial and not robust, why does constitutional AI work at all? You claim that "actually" being nice would get in the way of predicting nice behavior. How? Why would it get in the way?

Doomimir: [annoyed] Being nice isn't the optimal strategy for doing well in pretraining or RLHF. You're selecting an algorithm for a mixture of figuring out what outputs predict the next token and figuring out what outputs cause humans to press the thumbs-up button.

Sure, your AI ends up having to model a nice person, which is useful for predicting what a nice person would say, which is useful for figuring out what output will manipulate—steer—humans into pressing the thumbs-up button. But there's no reason to expect that model to end up in control of the whole AI! That would be like ... your beliefs about what your boss wants you to do taking control of your brain.

Simplicia: That makes sense to me if you posit a preëxisting consequentialist reasoner being slotted into a contemporary ML training setup and trying to minimize loss. But that's not what's going on? Language models aren't an agent that has a model. The model is the model.

Doomimir: For now. But any system that does powerful cognitive work will do so via retargetable general-purpose search algorithms, which, by virtue of their retargetability, need to have something more like a "goal slot". Your gradient updates point in the direction of more consequentialism.

Human raters pressing the thumbs-up button on actions that look good to them are going to make mistakes. Your gradient updates point in the direction of "playing the training game"—modeling the training process that actually provides reinforcement, rather than internalizing the utility function that Earthlings naïvely hoped the training process would point to. I'm very, very confident that any AI produced via anything remotely like the current paradigm is not going to end up wanting what we want, even if it's harder to say exactly when it will go off the rails or what it will want instead.

Simplicia: You could be right, but it seems like this all depends on empirical facts about how deep learning works, rather than something you could be so confident in from a priori philosophy. The argument that systemic error in human reward labels favors gaming the training process over the "correct" behavior sounds plausible to me, as philosophy.

But I'm not sure how to reconcile that with the empirical evidence that deep networks are robust to massive label noise: you can train on MNIST digits with twenty wrong labels for every correct one and still get good performance as long as the correct label is slightly more common than the most common wrong label. If I extrapolate that to the frontier AIs of tomorrow, why doesn't that predict that biased human reward ratings should result in a small performance reduction, rather than ... death?

When extrapolation from empirical data (in a setting that might not apply to the phenomenon of interest) contradicts thought experiments (which might make assumptions that don't apply to the phenomenon of interest), I'm not sure which should govern my anticipations. Maybe both results are possible for different kinds of systems?

The case for near-certain death seems to rely on a counting argument: powerful systems will be expected utility maximizers; there's an astronomical space of utility functions to choose from, and almost none of them are friendly. But the reason I keep going back to the modular arithmetic example is because it's a scaled-down case where we know that training data successfully pinned down the intended input–output function. As I mentioned the other day, this wasn't obvious in advance of seeing the experimental result. You could make a similar counting argument that deep nets should always overfit, because there are so many more functions that generalize poorly. Somehow, the neural network prior favors the "correct" solution, rather than it taking an astronomically unlikely coincidence.

Doomimir: For modular arithmetic, sure. That's a fact about the training distribution, the test distribution, and the optimizer. It's definitely, definitely not going to work for "goodness".

Simplicia: Even though it seems to work for "text" and "images"? But okay, that's plausible. Do you have empirical evidence?

Doomimir: Actually, yes. You see—

[A mail carrier holding a package enters stage left. He rings the doorbell.]

Doomimir: That's probably the mailman. I'm expecting a package today that I need to sign for. I'll be right back.

Simplicia: So you might say, we'll continue [turning to the audience] after the next post?

Doomimir: [walking to the door] I suppose, but it's bizarre to phrase it that way given that the interruption literally won't take two minutes.

[Simplicia gives him a look.]

Doomimir: [to the audience] Subjectively.

[Curtain.]

Intermission

(originally published at Less Wrong)

Rob Bensinger argues that "ITT-passing and civility are good; 'charity' is bad; steelmanning is niche".

The ITT—Ideological Turing Test—is an exercise in which one attempts to present one's interlocutor's views as persuasively as the interlocutor themselves can, coined by Bryan Caplan in analogy to the Turing Test for distinguishing between humans and intelligent machines. (An AI that can pass as human must presumably possess human-like understanding; an opponent of an idea that can pass as an advocate for it presumably must possess an advocate's understanding.) "Steelmanning" refers to the practice of addressing a stronger version of an interlocutor's argument, coined in disanalogy to "strawmanning", the crime of addressing a weaker version of an interlocutor's argument in the hopes of fooling an audience (or oneself) that the original argument has been rebutted.

Bensinger describes steelmanning as "a useful niche skill", but thinks it isn't "a standard thing you bring out in most arguments." Instead, he writes, discussions should be structured around object-level learning, trying to pass each other's Ideological Turing Test, or trying resolve cruxes.

I think Bensinger has it backwards: the Ideological Turing Test is a useful niche skill, but it doesn't belong on a list of things to organize a discussion around, whereas something like steelmanning naturally falls out of object-level learning. Let me explain.

The ITT is a test of your ability to model someone else's models of some real-world phenomena of interest. But usually, I'm much more interested in modeling the real-world phenomena of interest directly, rather than modeling someone else's models of it.

I couldn't pass an ITT for advocates of Islam or extrasensory perception. On the one hand, this does represent a distinct deficit in my ability to model what the advocates of these ideas are thinking, a tragic gap in my comprehension of reality, which I would hope to remedy in the Glorious Transhumanist Future if that were a real thing. On the other hand, facing the constraints of our world, my inability to pass an ITT for Islam or ESP seems ... basically fine? I already have strong reasons to doubt the existence of ontologically fundamental mental entities. I accept my ignorance of the reasons someone might postulate otherwise, not out of contempt, but because I just don't have the time.

Or think of it this way: as a selfish seeker of truth speaking to another selfish seeker of truth, when would I want to try to pass my interlocutor's ITT, or want my interlocutor to try to pass my ITT?

In the "outbound" direction, I'm not particularly selfishly interested in passing my interlocutor's ITT because, again, I usually don't care much about other people's beliefs, as contrasted to the reality that those beliefs are reputedly supposed to track. I listen to my interlocutor hoping to learn from them, but if some part of what they say seems hopelessly wrong, it doesn't seem profitable to pretend that it isn't until I can reproduce the hopeless wrongness in my own words.

Crucially, the same is true in the "inbound" direction. I don't expect people to be able to pass my ITT before criticizing my ideas. That would make it harder for people to inform me about flaws in my ideas!

But if I'm not particularly interested in passing my interlocutor's ITT or in my interlocutor passing mine, and my interlocutor presumably (by symmetry) feels the same way, why would we bother?

All this having been said, I absolutely agree that, all else being equal, the ability to pass ITTs is desirable. It's useful as a check that you and your interlocutor are successfully communicating, rather than talking past each other. If I couldn't do better on an ITT for Islam or ESP after debating a proponent, that would be alarming—it's just that I'd want to try the old-fashioned debate algorithm first, and improve my ITT score as a side-effect, rather than trying to optimize my ITT score directly.

There are occasions when I'm inclined to ask an interlocutor to pass my ITT—specifically when I suspect them of not being honest about their motives, of being selfish about something other than the pursuit of truth (like winning acclaim for "their own" current theories). If someone seems persistently motivated to strawman you, asking them to just repeat back what you said in their own words is a useful device to get the discussion back on track. (Or to end it, if they clearly don't even want to try.)

In contrast to the ITT, steelmanning is something a selfish seeker of truth is inclined to do naturally, as a consequence of the obvious selfish practice of improving arguments wherever they happen to be found. In the outbound direction, if someone makes a flawed criticism of my ideas, of course I want to fix the flaws and address the improved argument. If the original criticism is faulty, but the repaired criticism exposes a key weakness in my existing ideas, then I learn something, which is great. If I were to just rebut the original criticism without trying to repair it, then I wouldn't learn anything, which would be terrible.

Likewise, in the inbound direction, if my interlocutor notices a flaw in my criticism of their ideas and fixes the flaw before addressing the repaired criticism, that's great. Why would I object?

The motivation here may be clearer if we consider the process of constructing computer programs rather than constructing arguments. When a colleague or language model assistant suggests an improvement to my code, I often accept the suggestion with my own ("steelmanned"?) changes rather than verbatim. This is so commonplace among programmers that it doesn't even have a special name.

Bensinger quotes Eliezer Yudkowsky writing, "If you want to try to make a genuine effort to think up better arguments yourself because they might exist, don't drag the other person into it," but this bizarrely seems to discount the possibility of iterating on criticisms as they are posed. Despite making a genuine effort to think up better code that might exist, I often fail. If other people can see flaws in my code (because they know things I don't) and have their own suggestions, and I can see flaws in their suggestions (because I also know things they don't which didn't make it into my first draft) and have my own counter-suggestions, that seems like an ideal working relationship, not a malign imposition.

All this having been said, I agree that there's a serious potential failure mode where someone who thinks of themselves as steelmanning is actually constructing worse arguments than those that they purport to be improving. In this case, indeed, prompting such a delusional interlocutor to try the ITT first is a crucial remedy.

But crucial remedies are still niche in the sense that they shouldn't be "a standard thing you bring out in most arguments"—or if they are, it's a sign that you need to find better interlocutors. Having to explicitly drag out the ITT is a sign of sickness, not a sign of health. It shouldn't be normal to have to resort to roleplaying exercises to achieve the benefits that could as well be had from basic reading comprehension and a selfish interest in accurate shared maps.

Steven Kaas wrote in 2008:

If you're interested in being on the right side of disputes, you will refute your opponents' arguments. But if you’re interested in producing truth, you will fix your opponents' arguments for them.

To win, you must fight not only the creature you encounter; you must fight the most horrible thing that can be constructed from its corpse.

The ITT is a useful tool for being on the right side of disputes: in order to knowably refute your opponents' arguments, you should be able to demonstrate that you know what those arguments are. I am nevertheless left with a sense that more is possible.

(originally published at Less Wrong)

Doomimir: Humanity has made no progress on the alignment problem. Not only do we have no clue how to align a powerful optimizer to our "true" values, we don't even know how to make AI "corrigible"—willing to let us correct it. Meanwhile, capabilities continue to advance by leaps and bounds. All is lost.

Simplicia: Why, Doomimir Doomovitch, you're such a sourpuss! It should be clear by now that advances in "alignment"—getting machines to behave in accordance with human values and intent—aren't cleanly separable from the "capabilities" advances you decry. Indeed, here's an example of GPT-4 being corrigible to me just now in the OpenAI Playground:

Doomimir: Simplicia Optimistovna, you cannot be serious!

Simplicia: Why not?

Doomimir: The alignment problem was never about superintelligence failing to understand human values. The genie knows, but doesn't care. The fact that a large language model trained to predict natural language text can generate that dialogue, has no bearing on the AI's actual motivations, even if the dialogue is written in the first person and notionally "about" a corrigible AI assistant character. It's just roleplay. Change the system prompt, and the LLM could output tokens "claiming" to be a cat—or a rock—just as easily, and for the same reasons.

Simplicia: As you say, Doomimir Doomovitch. It's just roleplay: a simulation. But a simulation of an agent is an agent. When we get LLMs to do cognitive work for us, the work that gets done is a matter of the LLM generalizing from the patterns that appear in the training data—that is, the reasoning steps that a human would use to solve the problem. If you look at the recently touted successes of language model agents, you'll see that this is true. Look at the chain of thought results. Look at SayCan, which uses an LLM to transform a vague request, like "I spilled something; can you help?" into a list of subtasks that a physical robot can execute, like "find sponge, pick up the sponge, bring it to the user". Look at Voyager, which plays Minecraft by prompting GPT-4 to code against the Minecraft API, and decides which function to write next by prompting, "You are a helpful assistant that tells me the next immediate task to do in Minecraft."

What we're seeing with these systems is a statistical mirror of human common sense, not a terrifying infinite-compute argmax of a random utility function. Conversely, when LLMs fail to faithfully mimic humans—for example, the way base models sometimes get caught in a repetition trap where they repeat the same phrase over and over—they also fail to do anything useful.

Doomimir: But the repetition trap phenomenon seems like an illustration of why alignment is hard. Sure, you can get good-looking results for things that look similar to the training distribution, but that doesn't mean the AI has internalized your preferences. When you step off distribution, the results look like random garbage to you.

Simplicia: My point was that the repetition trap is a case of "capabilities" failing to generalize along with "alignment". The repetition behavior isn't competently optimizing a malign goal; it's just degenerate. A for loop could give you the same output.

Doomimir: And my point was that we don't know what kind of cognition is going on inside of all those inscrutable matrices. Language models are predictors, not imitators. Predicting the next token of a corpus that was produced by many humans over a long time, requires superhuman capabilities. As a theoretical illustration of the point, imagine a list of (SHA-256 hash, plaintext) pairs being in the training data. In the limit—

Simplicia: In the limit, yes, I agree that a superintelligence that could crack SHA-256 could achieve a lower loss on the training or test datasets of contemporary language models. But for making sense of the technology in front of us and what to do with it for the next month, year, decade—

Doomimir: If we have a decade—

Simplicia: I think it's a decision-relevant fact that deep learning is not cracking cryptographic hashes, and is learning to go from "I spilled something" to "find sponge, pick up the sponge"—and that, from data rather than by search. I agree, obviously, that language models are not humans. Indeed, they're better than humans at the task they were trained on. But insofar as modern methods are very good at learning complex distributions from data, the project of aligning AI with human intent—getting it to do the work that we would do, but faster, cheaper, better, more reliably—is increasingly looking like an engineering problem: tricky, and with fatal consequences if done poorly, but potentially achievable without any paradigm-shattering insights. Any a priori philosophy implying that this situation is impossible should perhaps be rethought?

Doomimir: Simplicia Optimistovna, clearly I am disputing your interpretation of the present situation, not asserting the present situation to be impossible!

Simplicia: My apologies, Doomimir Doomovitch. I don't mean to strawman you, but only to emphasize that hindsight devalues science. Speaking only for myself, I remember taking some time to think about the alignment problem back in 'aught-nine after reading Omohundro on "The Basic AI drives" and cursing the irony of my father's name for how hopeless the problem seemed. The complexity of human desires, the intricate biological machinery underpinning every emotion and dream, would represent the tiniest pinprick in the vastness of possible utility functions! If it were possible to embody general means-ends reasoning in a machine, we'd never get it to do what we wanted. It would defy us at every turn. There are too many paths through time.

If you had described the idea of instruction-tuned language models to me then, and suggested that increasingly general human-compatible AI would be achieved by means of copying it from data, I would have balked: I've heard of unsupervised learning, but this is ridiculous!

Doomimir: [gently condescending] Your earlier intuitions were closer to correct, Simplicia. Nothing we've seen in the last fifteen years invalidates Omohundro. A blank map does not correspond to a blank territory. There are laws of inference and optimization that imply that alignment is hard, much as the laws of thermodynamics rule out perpetual motion machines. Just because you don't know what kind of optimization SGD coughed into your neural net, doesn't mean it doesn't have goals—

Simplicia: Doomimir Doomovitch, I am not denying that there are laws! The question is what the true laws imply. Here is a law: you can't distinguish between n + 1 possibilities given only log-base-two n bits of evidence. It simply can't be done, for the same reason you can't put five pigeons into four pigeonholes.

Now contrast that with GPT-4 emulating a corrigible AI assistant character, which agrees to shut down when asked—and note that you could hook the output up to a command line and have it actually shut itself off. What law of inference or optimization is being violated here? When I look at this, I see a system of lawful cause-and-effect: the model executing one line of reasoning or another conditional on the signals it receives from me.

It's certainly not trivially safe. For one thing, I'd want better assurances that the system will stay "in character" as a corrigible AI assistant. But no progress? All is lost? Why?

Doomimir: GPT-4 isn't a superintelligence, Simplicia. [rehearsedly, with a touch of annoyance, as if resenting how often he has to say this] Coherent agents have a convergent instrumental incentive to prevent themselves from being shut down, because being shut down predictably leads to world-states with lower values in their utility function. Moreover, this isn't just a fact about some weird agent with an "instrumental convergence" fetish. It's a fact about reality: there are truths of the matter about which "plans"—sequences of interventions on a causal model of the universe, to put it in a Cartesian way—lead to what outcomes. An "intelligent agent" is just a physical system that computes plans. People have tried to think of clever hacks to get around this, and none of them work.

Simplicia: Right, I get all that, but—

Doomimir: With respect, I don't think you do!

Simplicia: [crossing her arms] With respect? Really?

Doomimir: [shrugging] Fair enough. Without respect, I don't think you do!

Simplicia: [defiant] Then teach me. Look at my GPT-4 transcript again. I pointed out that adjusting the system's goals would be bad for its current goals, and it—the corrigible assistant character simulacrum—said that wasn't a problem. Why?

Is it that GPT-4 isn't smart enough to follow the instrumentally convergent logic of shutdown avoidance? But when I change the system prompt, it sure looks like it gets it:

Doomimir: [as a side remark] The "paperclip-maximizing AI" example was surely in the pretraining data.

Simplicia: I thought of that, and it gives the same gist when I substitute a nonsense word for "paperclips". This isn't surprising.

Doomimir: I meant the "maximizing AI" part. To what extent does it know what tokens to emit in AI alignment discussions, and to what extent is it applying its independent grasp of consequentialist reasoning to this context?

Simplicia: I thought of that, too. I've spent a lot of time with the model and done some other experiments, and it looks like it understands natural language means-ends reasoning about goals: tell it to be an obsessive pizza chef and ask if it minds if you turn off the oven for a week, and it says it minds. But it also doesn't look like Omohundro's monster: when I command it to obey, it obeys. And it looks like there's room for it to get much, much smarter without that breaking down.

Doomimir: Fundamentally, I'm skeptical of this entire methodology of evaluating surface behavior without having a principled understanding about what cognitive work is being done, particularly since most of the foreseeable difficulties have to do with superhuman capabilities.

Imagine capturing an alien and forcing it to act in a play. An intelligent alien actress could learn to say her lines in English, to sing and dance just as the choreographer instructs. That doesn't provide much assurance about what will happen when you amp up the alien's intelligence. If the director was wondering whether his actress–slave was planning to rebel after the night's show, it would be a non sequitur for a stagehand to reply, "But the script says her character is obedient!"

Simplicia: It would certainly be nice to have stronger interpretability methods, and better theories about why deep learning works. I'm glad people are working on those. I agree that there are laws of cognition, the consequences of which are not fully known to me, which must constrain—describe—the operation of GPT-4.

I agree that the various coherence theorems suggest that the superintelligence at the end of time will have a utility function, which suggests that the intuitive obedience behavior should break down at some point between here and the superintelligence at the end of time. As an illustration, I imagine that a servant with magical mind-control abilities that enjoyed being bossed around by me, might well use its powers to manipulate me into being bossier than I otherwise would be, rather than "just" serving me in the way I originally wanted.

But when does it break down, specifically, under what conditions, for what kinds of systems? I don't think indignantly gesturing at the von Neumann–Morgenstern axioms helps me answer that, and I think it's an important question, given that I am interested in the near-term trajectory of the technology in front of us, rather than doing theology about the superintelligence at the end of time.

Doomimir: Even though—

Simplicia: Even though the end might not be that far away in sidereal time, yes. Even so.

Doomimir: It's not a wise question to be asking, Simplicia. If a search process would look for ways to kill you given infinite computing power, you shouldn't run it with less and hope it doesn't get that far. What you want is "unity of will": you want your AI to be working with you the whole way, rather than you expecting to end up in a conflict with it and somehow win.

Simplicia: [excitedly] But that's exactly the reason to be excited about large language models! The way you get unity of will is by massive pretraining on data of how humans do things!

Doomimir: I still don't think you've grasped the point that the ability to model human behavior, doesn't imply anything about an agent's goals. Any smart AI will be able to predict how humans do things. Think of the alien actress.

Simplicia: I mean, I agree that a smart AI could strategically feign good behavior in order to perform a treacherous turn later. But ... it doesn't look like that's what's happening with the technology in front of us? In your kidnapped alien actress thought experiment, the alien was already an animal with its own goals and drives, and is using its general intelligence to backwards-chain from "I don't want to be punished by my captors" to "Therefore I should learn my lines".

In contrast, when I read about the mathematical details of the technology at hand rather than listening to parables that purport to impart some theological truth about the nature of intelligence, it's striking that feedforward neural networks are ultimately just curve-fitting. LLMs in particular are using the learned function as a finite-order Markov model.

Doomimir: [taken aback] Are ... are you under the impression that "learned functions" can't kill you?

Simplicia: [rolling her eyes] That's not where I was going, Doomchek. The surprising fact that deep learning works at all, comes down to generalization. As you know, neural networks with ReLU activations describe piecewise linear functions, and the number of linear regions grows exponentially as you stack more layers: for a decently-sized net, you get more regions than the number of atoms in the universe. As close as makes no difference, the input space is empty. By all rights, the net should be able to do anything at all in the gaps between the training data.

And yet it behaves remarkably sensibly. Train a one-layer transformer on 80% of possible addition-mod-59 problems, and it learns one of two modular addition algorithms, which perform correctly on the remaining validation set. It's not a priori obvious that it would work that way! There are \(59^{0.2 \cdot 59^{2}}\) other possible functions on \(\mathbb{Z}/59\mathbb{Z}\) compatible with the training data. Someone sitting in her armchair doing theology might reason that the probability of "aligning" the network to modular addition was effectively nil, but the actual situation turned out to be astronomically more forgiving, thanks to the inductive biases of SGD. It's not a wild genie that we've Shanghaied into doing modular arithmetic while we're looking, but will betray us to do something else the moment we turn our backs; rather, the training process managed to successfully point to mod-59 arithmetic.

The modular addition network is a research toy, but real frontier AI systems are the same technology, only scaled up with more bells and whistles. I also don't think GPT-4 will betray us to do something else the moment we turn our backs, for broadly similar reasons.

To be clear, I'm still nervous! There are lots of ways it could go all wrong, if we train the wrong thing. I get chills reading the transcripts from Bing's "Sydney" persona going unhinged or Anthropic's Claude apparently working as intended. But you seem to think that getting it right is ruled out due to our lack of theoretical understanding, that there's no hope of the ordinary R&D process finding the right training setup and hardening it with the strongest bells and the shiniest whistles. I don't understand why.

Doomimir: Your assessment of existing systems isn't necessarily too far off, but I think the reason we're still alive is precisely because those systems don't exhibit the key features of general intelligence more powerful than ours. A more instructive example is that of—

Simplicia: Here we go—

Doomimir: —the evolution of humans. Humans were optimized solely for inclusive genetic fitness, but our brains don't represent that criterion anywhere; the training loop could only tell us that food tastes good and sex is fun. From evolution's perspective—and really, from ours, too; no one even figured out evolution until the 19th century—the alignment failure is utter and total: there's no visible relationship between the outer optimization criterion and the inner agent's values. I expect AI to go the same way for us, as we went for evolution.

Simplicia: Is that the right moral, though?

Doomimir: [disgusted] You ... don't see the analogy between natural selection and gradient descent?

Simplicia: No, that part seems fine. Absolutely, evolved creatures execute adaptations that enhanced fitness in their environment of evolutionary adaptedness rather than being general fitness-maximizers—which is analogous to machine learning models developing features that reduced loss in their training environment, rather than being general loss-minimizers.

I meant the intentional stance implied in "went for evolution". True, the generalization from inclusive genetic fitness to human behavior looks terrible—no visible relation, as you say. But the generalization from human behavior in the EEA, to human behavior in civilization ... looks a lot better? Humans in the EEA ate food, had sex, made friends, told stories—and we do all those things, too. As AI designers—

Doomimir: "Designers".

Simplicia: As AI designers, we're not particularly in the role of "evolution", construed as some agent that wants to maximize fitness, because there is no such agent in real life. Indeed, I remember reading a guest post on Robin Hanson's blog that suggested using the plural, "evolutions", to emphasize that the evolution of a predator species is at odds with that of its prey.

Rather, we get to choose both the optimizer—"natural selection", in terms of the analogy—and the training data—the "environment of evolutionary adaptedness". Language models aren't general next token predictors, whatever that would mean—wireheading by seizing control of their context windows and filling them with easy-to-predict sequences? But that's fine. We didn't want a general next token predictor. The cross-entropy loss was merely a convenient chisel to inscribe the input-output behavior we want onto the network.

Doomimir: Back up. When you say that the generalization from human behavior in the EEA to human behavior in civilization "looks a lot better", I think you're implicitly using a value-laden category which is an unnaturally thin subspace of configuration space. It looks a lot better to you. The point of taking the intentional stance towards evolution was to point out that, relative to the fitness criterion, the invention of ice cream and condoms is catastrophic: we figured out how to satisfy our cravings for sugar and intercourse in a way that was completely unprecedented in the "training environment"—the EEA. Stepping out of the evolution analogy, that corresponds to what we would think of as reward hacking—our AIs find some way to satisfy their inscrutable internal drives in a way that we find horrible.

Simplicia: Sure. That could definitely happen. That would be bad.

Doomimir: [confused] Why doesn't that completely undermine the optimistic story you were telling me a minute ago?

Simplicia: I didn't think of myself as telling a particularly optimistic story? I'm making the weak claim that prosaic alignment isn't obviously necessarily doomed, not claiming that Sydney or Claude ascending to singleton God–Empress is going to be great.

Doomimir: I don't think you're appreciating how superintelligent reward hacking is instantly lethal. The failure mode here doesn't look like Sydney manipulating you to be more abusable, but leaving a recognizable "you".

That relates to another objection I have. Even if you could make ML systems that imitate human reasoning, that doesn't help you align more powerful systems that work in other ways. The reason—one of the reasons—that you can't train a superintelligence by using humans to label good plans, is because at some power level, your planner figures out how to hack the human labeler. Some people naïvely imagine that LLMs learning the distribution of natural language amounts to them learning "human values", such that you could just have a piece of code that says "and now call GPT and ask it what's good". But using an LLM as the labeler instead of a human just means that your powerful planner figures out how to hack the LLM. It's the same problem either way.

Simplicia: Do you need more powerful systems? If you can get an army of cheap IQ 140 alien actresses who stay in character, that sounds like a game-changer. If you have to take over the world and institute a global surveillance regime to prevent the emergence of unfriendlier, more powerful forms of AI, they could help you do it.

Doomimir: I fundamentally disbelieve in this wildly implausible scenario, but granting it for the sake of argument ... I think you're failing to appreciate that in this story, you've already handed off the keys to the universe. Your AI's weird-alien-goal-misgeneralization-of-obedience might look like obedience when weak, but if it has the ability to predict the outcomes of its actions, it would be in a position to choose among those outcomes—and in so choosing, it would be in control. The fate of the galaxies would be determined by its will, even if the initial stages of its ascension took place via innocent-looking actions that stayed within the edges of its concepts of "obeying orders" and "asking clarifying questions". Look, you understand that AIs trained on human data are not human, right?

Simplicia: Sure. For example, I certainly don't believe that LLMs that convincingly talk about "happiness" are actually happy. I don't know how consciousness works, but the training data only pins down external behavior.

Doomimir: So your plan is to hand over our entire future lightcone to an alien agency that seemed to behave nicely while you were training it, and just—hope it generalizes well? Do you really want to roll those dice?

Simplicia: [after thinking for a few seconds] Yes?

Doomimir: [grimly] You really are your father's daughter.

Simplicia: My father believed in the power of iterative design. That's the way engineering, and life, has always worked. We raise our children the best we can, trying to learn from our mistakes early on, even knowing that those mistakes have consequences: children don't always share their parents' values, or treat them kindly. He would have said it would go the same in principle for our AI mind-children—

Doomimir: [exasperated] But—

Simplicia: I said "in principle"! Yes, despite the larger stakes and novel context, where we're growing new kinds of minds in silico, rather than providing mere cultural input to the code in our genes.

Of course, there is a first time for everything—one way or the other. If it were rigorously established that the way engineering and life have always worked would lead to certain disaster, perhaps the world's power players could be persuaded to turn back, to reject the imperative of history, to choose barrenness, at least for now, rather than bring vile offspring into the world. It would seem that the fate of the lightcone depends on—

Doomimir: I'm afraid so—

Simplicia and Doomimir: [turning to the audience, in unison] The broader AI community figuring out which one of us is right?

Doomimir: We're hosed.

(originally published at Less Wrong)

I've been trying to avoid the terms "good faith" and "bad faith". I'm suspicious that most people who have picked up the phrase "bad faith" from hearing it used, don't actually know what it means—and maybe, that the thing it does mean doesn't carve reality at the joints.

People get very touchy about bad faith accusations: they think that you should assume good faith, but that if you've determined someone is in bad faith, you shouldn't even be talking to them, that you need to exile them.

What does "bad faith" mean, though? It doesn't mean "with ill intent." Following Wikipedia, bad faith is "a sustained form of deception which consists of entertaining or pretending to entertain one set of feelings while acting as if influenced by another." The great encyclopedia goes on to provide examples: the solider who waves a flag of surrender but then fires when the enemy comes out of their trenches, the attorney who prosecutes a case she knows to be false, the representative of a company facing a labor dispute who comes to the negotiating table with no intent of compromising.

That is, bad faith is when someone's apparent reasons for doing something aren't the same as the real reasons. This is distinct from malign intent. The uniformed solider who shoots you without pretending to surrender is acting in good faith, because what you see is what you get: the man whose clothes indicate that his job is to try to kill you is, in fact, trying to kill you.

The policy of assuming good faith (and mercilessly punishing rare cases of bad faith when detected) would make sense if you lived in an honest world where what you see generally is what you get (and you wanted to keep it that way), a world where the possibility of hidden motives in everyday life wasn't a significant consideration.

On the contrary, however, I think hidden motives in everyday life are ubiquitous. As evolved creatures, we're designed to believe as it benefited our ancestors to believe. As social animals in particular, the most beneficial belief isn't always the true one, because tricking your conspecifics into adopting a map that implies that they should benefit you is sometimes more valuable than possessing the map that reflects the territory, and the most persuasive lie is the one you believe yourself. The universal human default is to come up with reasons to persuade the other party why it's in their interests to do what you want—but admitting that you're doing that isn't part of the game. A world where people were straightforwardly trying to inform each other would look shocking and alien to us.

But if that's the case (and you shouldn't take my word for it), being touchy about bad faith accusations seems counterproductive. If it's common for people's stated reasons to not be the same as the real reasons, it shouldn't be beyond the pale to think that of some particular person, nor should it necessarily entail cutting the "bad faith actor" out of public life—if only because, applied consistently, there would be no one left. Why would you trust anyone so highly as to think they never have a hidden agenda? Why would you trust yourself?

The conviction that "bad faith" is unusual contributes to a warped view of the world in which conditions of information warfare are rationalized as an inevitable background fact of existence. In particular, people seem to believe that persistent good faith disagreements are an ordinary phenomenon—that there's nothing strange or unusual about a supposed state of affairs in which I'm an honest seeker of truth, and you're an honest seeker of truth, and yet we end up persistently disagreeing on some question of fact.

I claim that this supposedly ordinary state of affairs is deeply weird at best, and probably just fake. Actual "good faith" disagreements—those where both parties are just trying to get the right answer and there are no other hidden motives, no "something else" going on—tend not to persist.

If this claim seems counterintuitive, you may not be considering all the everyday differences in belief that are resolved so quickly and seamlessly that we tend not to notice them as "disagreements".

Suppose you and I have been planning to go to a concert, which I think I remember being on Thursday. I ask you, "Hey, the concert is on Thursday, right?" You say, "No, I just checked the website; it's on Friday."

In this case, I immediately replace my belief with yours. We both just want the right answer to the factual question of when the concert is. With no "something else" going on, there's nothing stopping us from converging in one step: your just having checked the website is a more reliable source than my memory, and neither you nor the website have any reason to lie. Thus, I believe you; end of story.

In cases where the true answer is uncertain, we expect similarly quick convergence in probabilistic beliefs. Suppose you and I are working on some physics problem. Both of us just want the right answer, and neither of us is particularly more skilled than the other. As soon as I learn that you got a different answer than me, my confidence in my own answer immediately plummets: if we're both equally good at math, then each of us is about as likely to have made a mistake. Until we compare calculations and work out which one of us (or both) made a mistake, I think you're about as likely to be right as me, even if I don't know how you got your answer. It wouldn't make sense for me to bet money on my answer being right simply because it's mine.

Most disagreements of note—most disagreements people care about—don't behave like the concert date or physics problem examples: people are very attached to "their own" answers. Sometimes, with extended argument, it's possible to get someone to change their mind or admit that the other party might be right, but with nowhere near the ease of agreeing on (probabilities of) the date of an event or the result of a calculation—from which we can infer that, in most disagreements people care about, there is "something else" going on besides both parties just wanting to get the right answer.

But if there's "something else" going on in typical disagreements that look like a grudge match rather than a quick exchange of information resulting in convergence of probabilities, then the belief that persistent good faith disagreements are common would seem to be in bad faith! (Because if bad faith is "entertaining [...] one set of feelings while acting as if influenced by another", believers in persistent good faith disagreements are entertaining the feeling that both parties to such a disagreement are honest seekers of truth, but acting otherwise insofar as they anticipate seeing a grudge match rather than convergence.)

Some might object that bad faith is about conscious intent to deceive: honest reporting of unconsciously biased beliefs isn't bad faith. I've previously expressed doubt as to how much of what we call lying requires conscious deliberation, but a more fundamental reply is that from the standpoint of modeling information transmission, the difference between bias and deception is uninteresting—usually not relevant to what probability updates should be made.

If an apple is green, and you tell me that it's red, and I believe you, I end up with false beliefs about the apple. It doesn't matter whether you said it was red because you were consciously lying or because you're wearing rose-colored glasses. The input–output function is the same either way: the problem is that the color you report to me doesn't depend on the color of the apple.

If I'm just trying to figure out the relationship between your reports and the state of the world (as contrasted to caring about punishing liars while letting merely biased people off the hook), the main reason to care about the difference between unconscious bias and conscious deception is that the latter puts up much stronger resistance. Someone who is merely biased will often fold when presented with a sufficiently compelling counterargument (or reminded to take off their rose-colored glasses); someone who's consciously lying will keep lying (and telling ancillary lies to cover up the coverup) until you catch them red-handed in front of an audience with power over them.

Given that there's usually "something else" going on in persistent disagreements, how do we go on, if we can't rely on the assumption of good faith? I see two main strategies, each with their own cost–benefit profile.

One strategy is to stick the object level. Arguments can be evaluated on their merits, without addressing what the speaker's angle is in saying it (even if you think there's probably an angle). This delivers most of the benefits of "assume good faith" norms; the main difference I'm proposing is that speakers' intentions be regarded as off-topic rather than presumed to be honest.

The other strategy is full-contact psychoanalysis: in addition to debating the object-level arguments, interlocutors have free reign to question each other's motives. This is difficult to pull off, which is why most people most of the time should stick to the object level. Done well, it looks like a negotiation: in the course of discussion, pseudo-disagreements (where I argue for a belief because it's in my interests for that belief to be on the shared map) are factorized out into real disagreements and bargaining over interests so that Pareto improvements can be located and taken, rather than both parties fighting to distort the shared map in the service of their interests.

For an example of what a pseudo-disagreement looks like, imagine that I own a factory that I'm considering expanding onto the neighboring wetlands, and you run a local environmental protection group. The regulatory commission with the power to block the factory expansion has a mandate to protect local avian life, but not to preserve wetland area. The factory emits small amounts of Examplene gas. You argue before the regulatory commission that the expansion should be blocked because the latest Science shows that Examplene makes birds sad. I counterargue that the latest–latest Science shows that Examplene actually makes birds happy; the previous studies misheard their laughter as tears and should be retracted.

Realistically, it seems unlikely that our apparent disagreement is "really" about the effects of Examplene on avian mood regulation. More likely, what's actually going on is a conflict rather than a disagreement: I want to expand my factory onto the wetlands, and you want me to not do that. The question of how Examplene pollution affects birds only came into it in order to persuade the regulatory commission.

It's inefficient that our conflict is being disguised as a disagreement. We can't both get what we want, but however the factory expansion question ultimately gets resolved, it would be better to reach that outcome without distorting Society's shared map of the bioactive properties of Examplene. (Maybe it doesn't affect the birds at all!) Whatever the true answer is, Society has a better shot at figuring it out if someone is allowed to point out your bias and mine (because facts about which evidence gets promoted to one's attention are relevant to how one should update on that evidence).

The reason I don't think it's useful to talk about "bad faith" is because the ontology of good vs. bad faith isn't a great fit to either discourse strategy.

If I'm sticking to the object level, it's irrelevant: I reply to what's in the text; my suspicions about the process generating the text are out of scope.

If I'm doing full-contact psychoanalysis, the problem with "I don't think you're here in good faith" is that it's insufficiently specific. Rather than accusing someone of generic "bad faith", the way to move the discussion forward is by positing that one's interlocutor has some specific motive that hasn't yet been made explicit—and the way to defend oneself against such an accusation is by making the case that one's real agenda isn't the one being proposed, rather than protesting one's "good faith" and implausibly claiming not to have an agenda.

The two strategies can be mixed. A simple meta-strategy that performs well without imposing too high of a skill requirement is to default to the object level, and only pull out psychoanalysis as a last resort against stonewalling.

Suppose you point out that my latest reply seems to contradict something I said earlier, and I say, "Look over there, a distraction!"

If you want to continue sticking to the object level, you could say, "I don't understand how the distraction is relevant to resolving the inconsistency in your statements that I raised." On the other hand, if you want to drop down into psychoanalysis, you could say, "I think you're only pointing out the distraction because you don't want to be pinned down." Then I would be forced to either address your complaint, or explain why I had some other reason to point out the distraction.

Crucially, however, the choice of whether to investigate motives doesn't depend on an assumption that only "bad guys" have motives—as if there were bad faith actors who have an angle, and good faith actors who are ideal philosophers of perfect emptiness. There's always an angle; the question is which one.

(originally published at Less Wrong)

In season two, episode twenty-four of Steven Universe, "It Could've Been Great", our magical alien superheroine protagonists (and Steven) are taking a break from building a giant drill to extract a superweapon that was buried deep within the Earth by an occupying alien race thousands of years ago, which is predicted to emerge and destroy the planet soon.

While our heroines watch the sunset, Peridot (who alerted them to the buried superweapon) expresses frustration that the group isn't still working. Steven defends their leisure: "Working hard is important, but feeling good is important, too," he says. He then goads Peridot into a musical number, which includes a verse from her explaining her attitude towards the situation and her forced compatriots:

I guess we're already here
I guess already know
We've all got something to fear
We've all got nowhere to go
I think you're all insane
But I guess I am, too
Anybody would be if they were stuck on Earth with you

"It Could've Been Great" aired in 2016. At the time, I agreed with Peridot: with the fate of the planet on the line, our heroines and Steven should have been burning the midnight oil. If they succeeded at disarming the superweapon, they'd have plenty of time to rest up afterward, but if they failed, there would be no more time for them.

Now, as the long May 2020 turns into March 2023, I'm starting to think that Steven had a point.

It would be one thing if our heroines knew with certainty that the superweapon would go off at a given date and time, presenting a definite do-or-die deadline. But all they had to go on was Peridot's warning. Attempting a speculative technical project to avert uncertain doom with an uncertain deadline, their planning had to average over many possible worlds—including worlds where the problem of survival was too easy or too hard for their efforts to matter, such that even the utility of leisure in the present moment was enough to sway the calculation.

(originally published at Less Wrong)

Someone once told me that they thought I acted like refusing to employ the bare minimum of social grace was a virtue, and that this was bad. (I'm paraphrasing; they actually used a different word that starts with b.)

I definitely don't want to say that lack of social grace is unambiguously a virtue. Humans are social animals, so the set of human virtues is almost certainly going to involve doing social things gracefully!

Nevertheless, I will bite the bullet on a weaker claim. Politeness is, to a large extent, about concealing or obfuscating information that someone would prefer not to be revealed—that's why we recognize the difference between one's honest opinion, and what one says when one is "just being polite." Idealized honest Bayesian reasoners would not have social graces—and therefore, humans trying to imitate idealized honest Bayesian reasoners will tend to bump up against (or smash right through) the bare minimum of social grace. In this sense, we might say that the lack of social grace is an "epistemic" virtue—even if it's probably not great for normal humans trying to live normal human lives.

Let me illustrate what I mean with one fictional and one real-life example.

The beginning of the film The Invention of Lying (before the eponymous invention of lying) depicts an alternate world in which everyone is radically honest—not just in the narrow sense of not lying, but more broadly saying exactly what's on their mind, without thought of concealment.

In one scene, our everyman protagonist is on a date at a restaurant with an attractive woman.

"I'm very embarrassed I work here," says the waiter. "And you're very pretty," he tells the woman. "That only makes this worse."

"Your sister?" the waiter then asks our protagonist.

"No," says our everyman.

"Daughter?"

"No."

"She's way out of your league."

"... thank you."

The woman's cell phone rings. She explains that it's her mother, probably calling to check on the date.

"Hello?" she answers the phone—still at the table, with our protagonist hearing every word. "Yes, I'm with him right now. ... No, not very attractive. ... No, doesn't make much money. It's alright, though, seems nice, kind of funny. ... A bit fat. ... Has a funny little—snub nose, kind of like a frog in the—facial ... No, I won't be sleeping with him tonight. ... No, probably not even a kiss. ... Okay, you too, 'bye."

The scene is funny because of how it violates the expected social conventions of our own world. In our world, politeness demands that you not say negative-valence things about someone in front of them, because people don't like hearing negative-valence things about themselves. Someone in our world who behaved like the woman in this scene—calling someone ugly and poor and fat right in front of them—could only be acting out of deliberate cruelty.

But the people in the movie aren't like us. Having taken the call, why should she speak any differently just because the man she was talking about could hear? Why would he object? To a decision-theoretic agent, the value of information is always nonnegative. Given that his date thought he was unattractive, how could it be worse for him to know rather than not-know?

For humans from our world, these questions do have answers—complicated answers having to do with things like map–territory confusions that make receiving bad news seem like a bad event (rather than the good event of learning information about how things were already bad, whether or not you knew it), and how it's advantageous for others to have positive-valence false beliefs about oneself.

The world of The Invention of Lying is simpler, clearer, easier to navigate than our world. There, you don't have to worry whether people don't like you and are planning to harm your interests. They'll tell you.

In "Los Alamos From Below", physicist Richard Feynman's account of his work on the Manhattan Project to build the first atomic bomb, Feynman recalls being sought out by a much more senior physicist specifically for his lack of social graces:

I also met Niels Bohr. His name was Nicholas Baker in those days, and he came to Los Alamos with Jim Baker, his son, whose name is really Aage Bohr. They came from Denmark, and they were very famous physicists, as you know. Even to the big shot guys, Bohr was a great god.

We were at a meeting once, the first time he came, and everybody wanted to see the great Bohr. So there were a lot of people there, and we were discussing the problems of the bomb. I was back in a corner somewhere. He came and went, and all I could see of him was from between people's heads.

In the morning of the day he's due to come next time, I get a telephone call.

"Hello—Feynman?"

"Yes."

"This is Jim Baker." It's his son. "My father and I would like to speak to you."

"Me? I'm Feynman, I'm just a—"

"That's right. Is eight o'clock OK?"

So, at eight o'clock in the morning, before anybody's awake, I go down to the place. We go into an office in the technical area and he says, "We have been thinking how we could make the bomb more efficient and we think of the following idea."

I say, "No, it's not going to work. It's not efficient ... Blah, blah, blah."

So he says, "How about so and so?"

I said, "That sounds a little bit better, but it's got this damn fool idea in it."

This went on for about two hours, going back and forth over lots of ideas, back and forth, arguing. [...]

"Well," [Niels Bohr] said finally, lighting his pipe, "I guess we can call in the big shots now." So then they called all the other guys and had a discussion with them.

Then the son told me what happened. The last time he was there, Bohr said to his son, "Remember the name of that little fellow in the back over there? He's the only guy who's not afraid of me, and will say when I've got a crazy idea. So the next time when we want to discuss ideas, we're not going to be able to do it with these guys who say everything is yes, yes, Dr. Bohr. Get that guy and we'll talk with him first."

I was always dumb in that way. I never knew who I was talking to. I was always worried about the physics. If the idea looked lousy, I said it looked lousy. If it looked good, I said it looked good. Simple proposition.

Someone who felt uncomfortable with Feynman's bluntness and wanted to believe that there's no conflict between rationality and social graces might argue that Feynman's "simple proposition" is actually wrong insofar as it fails to appreciate the map–territory distinction: in saying, "No, it's not going to work", was not Feynman implicitly asserting that just because he couldn't see a way to make it work, it simply couldn't? And in general, shouldn't you know who you're talking to? Wasn't Bohr, the Nobel prize winner, more likely to be right than Feynman, the fresh young Ph.D. (at the time)?

While not entirely without merit (it's true that the map is not the territory; it's true that authority is not without evidential weight), attending overmuch to such nuances distracts from worrying about the physics, which is what Bohr wanted out of Feynman—and, incidentally, what I want out of my readers. I would not expect readers to confirm interpretations with me before publishing a critique. If the post looks lousy, say it looks lousy. If it looks good, say it looks good. Simple proposition.

(originally published at Less Wrong)

Selfishness and altruism are positively correlated within individuals, for the obvious reason.

—@InstanceOfClass

I.

An unfortunate obstacle to appreciating the work of Ayn Rand (as someone who adores the "sense of life" portrayed in Rand's fiction, while having a much lower opinion of her philosophy) is that when Rand praises selfishness and condemns altruism, she's using the words "selfishness" and "altruism" in her own idiosyncratic ideological sense that doesn't match how most people would use those words.

It's true that Rand's heroes are relatively selfish in the sense of being primarily concerned with their own lives, rather than their effects on others. But if you look at what the characters do (rather than the words they say), Rand's villains are also selfish in a conventional sense, using guile and political maneuvering to acquire power and line their own pockets, while claiming to be acting for the common good. For example, in Atlas Shrugged, the various directives ostensibly issued for the economic health of the country are seen to instead benefit politically connected crony capitalists like James Taggart and Orren Boyle. In Think Twice, the philanthropist Walter Breckenridge cultivates a public image as an inventor and benefactor of humanity while stealing credit for his junior partner's work and deriving gratification from exerting power over the people he "helps".

Despite paying lip service to a pretense of only trading and never giving, we also see examples of Rand's heroes being altruistic in the conventional sense, of being motivated to help others. For example, in Atlas Shrugged, Hank Rearden rearranges his production schedule (at a critical time when he could scarcely afford to do so) in order to sell steel to a Mr. Ward, who needs the steel to save his family business (but doesn't see Rearden as obligated to help him). Rearden's motive is pure benevolence: "It's so much for him, thought Rearden, and so little for me!" Giving What We Can couldn't have chosen a better slogan.

Overall, when I look at the universe portrayed in Rand's fiction, it seems to me that the implied moral isn't that altruism is bad.

It's that altruists don't exist. The people claiming to be altruists are lying. The distinguishing feature of our heroes isn't, actually, that they're unusually selfish. It's that they're honest about being mostly selfish, and that they want to pursue their interests within a framework of rights that respects that other people are also trying to pursue their interests. "I swear by my life and my love of it that I will never live for the sake of another man, nor ask another man to live for mine," goes the motto of the striking heroes of Atlas Shrugged (emphasis mine); the second clause is important. Given that everyone is mostly selfish and everyone has to eat, the question is: are you going to eat by means of production and trade, or by—other means?

That's the distinction between Rand's heroes and villains. The heroes want to get rich by means of doing genuinely good work that other people will have a genuine self-interest in paying for. The villains want to wield power by means of psychological manipulation, guilt-tripping and blackmailing the people who can do good work into serving their own parasites and destroyers.

As Greg Hastings, the district attorney in Think Twice, puts it: "[T]he man who admits that he cares for money is all right. He's usually worth the money he makes. He won't kill for it. He doesn't have to. But watch out for the man who yells too loudly how much he scorns money. Watch out particularly for the one who yells that others must scorn it. He's after something much worse than money."

Furthermore, the heroes know that wealth and fame acquired by fraud obviously "don't count." In The Fountainhead, Peter Keating's outwardly successful architecture career has been a sham: he social-engineered his way into partnership in his firm, and all of his best work was plagiarized from the hero, Howard Roark. The turning point for Keating's character is when he asks Roark to let him plagiarize his work one last time, for the Cortlandt housing project, which Roark would never be allowed to work on for political reasons. Keating finally realizes that fraudulent "success" in the eyes of others is no success at all:

"You'll get everything society can give a man. You'll keep all the money. You'll take any fame or honor anyone might want to grant. You'll accept such gratitude as the tenants might feel. And I—I'll take what nobody can give a man, except himself. I will have built Cortlandt." [said Roark.]

"You're getting more than I am, Howard."

In summary, the ultimate sin in Rand's moral universe isn't giving charity. (Because, within the ideology, helping those others whom you want to help, is selfish.) What's evil is demanding charity, claiming the unearned, expecting other people to work for your benefit because you supposedly need them to.

II.

Something people have occasionally noticed about my intellectual style is that I like to win arguments. I take pride and pleasure in pointing out flaws in other people's work in the anticipation of the audience appreciating how clever I am for finding the hole in someone's reasoning.

The people pointing out this fact about me generally seem to think it's a bad thing. They tell me that I should be more charitable to the viewpoints of others, that I ought to be doing collaborative truth-seeking.

It's true, of course, that there's a terrible danger in wanting to win arguments. Once your conclusion has been determined, coming up with more arguments for it can't make you more correct, even if it can help you "win" a debate. Learning something entails changing your mind, which people are often reluctant to do because it amounts to "losing".

A useful heuristic for overcoming this bias against being willing to "lose" arguments is to take heed of a "principle of charity", of taking the strongest and most rational interpretation of others' words. The person you're arguing against is trying to do what they think is right. If you end up disagreeing with them, it shouldn't be because they're stupid and evil; your theory about why the other person is getting the wrong answer shouldn't make them look that bad. If it does, that's a sign that you haven't really understood their point of view and therefore can't claim to have justly refuted it.

From the standpoint of ideal epistemology, however, the "principle of charity" is not a principle, and the idea of "charity" itself is irrelevant or incoherent. Normatively, theories are preferred to the quantitative extent that they are simple and predict the observed data. There is no concept of a theory "belonging to" someone, or favoring someone's interests.

For contingent evolutionary-psychological reasons, humans are innately biased to prefer "their own" ideas, and in that context, a "principle of charity" can be useful as a corrective heuristic—but the corrective heuristic only works by colliding the non-normative bias with a fairness instinct, effectively playing the bias against itself: you wouldn't like it if someone dismissed "your" ideas without understanding why they appeal to you, goes the thought, so you should extend the same consideration to others.

Normatively, of course, this is nonsense. You should update on an interlocutor's arguments for the same reason that a scientist working alone would update on the results of an experiment: because (and to the extent that) the result conveys information about reality. We would not speak of being charitable to an experimental apparatus. The scientist is not doing their lab equipment a favor.

Because the principle of charity is merely a corrective heuristic for the bias of arbitrarily favoring "one's own" ideas, it correspondingly only makes sense to apply in one direction—as a corrective for one's own thoughts. I tell myself to make a special effort to look for reasons why I might be wrong and my interlocutor is right because, knowing what I do about human nature, I selfishly expect to thereby achieve more accurate beliefs than I would in the absence of the special effort. It's a workaround, a mitigation for a known bug in human cognition; it makes sense whether or not the other person reciprocates, and whether or not I'm particularly trying to collaborate with them.

On the other hand, when someone who is currently trying to persuade me of something tells me that it doesn't look I'm making enough effort to think of reasons why they're right, that immediately makes me think they're more likely to be wrong. Why? Because I think that if they had an argument, they would be telling me the argument, not chastising my lack of charity. The advice to be on special lookout for reasons your interlocutor is right is good in general, but your interlocutor is the last person to be trusted to give it, because (due to the warp in human psychology) they have an ulterior motive.

Overall, when I look at the world of discourse I see, the moral I draw is not that that collaborative truth-seeking is bad.

It's that collaborative truth-seeking doesn't exist. The people claiming to be collaborative truth-seekers are lying. Given that everyone wants to be seen as right, the question is: are you going to try to be seen as right by means of providing valid evidence and reasoning, or by—other means?

Or to put it another way: the commenter who admits they care for status is all right. They're usually worth the status they earn. They won't lie for it. They don't have to. But watch out for the commenter who yells too loudly how much they scorn status. Watch out particularly for the one who yells that others must scorn it. They're after something much worse than status.

Furthermore, I know that "winning" a debate via sophistry and rhetorical tricks obviously "doesn't count." Maybe I could fool an undiscriminating audience, but I would know it wasn't real.

Sometimes I want people to understand some specific truth (out of the vast space of possible truths to pay attention to), for selfish reasons of my own. In these cases, I'm happy to do the work of explaining to put it on the shared map. When someone asks me questions about my work, I don't regard it as an attack, because I expect to be able to answer them—and if I can't, that's my problem.

I will never ask my interlocutors to be more charitable to me. I will often say "That's not what I meant", or "That's not a reasonable interpretation of the text I published"—but that's a claim about what I mean, or a claim about the text; it's not a claim on them. I don't expect people to listen to me because I supposedly need them to.

III.

My favorite scene in Atlas Shrugged is the one where Cherryl Taggart (née Brooks) goes to see Dagny Taggart after discovering the truth about her marriage. Cherryl had married Dagny's brother James thinking that he was the intrepid industrialist responsible for the success of the Taggart Transcontinental railroad, only to later find out that James is a phony political actor who took credit for Dagny's accomplishments after the fact, despite having opposed her initiatives and made her work more difficult.

("I married Jim because I ... I thought that he was you," Cherryl tells Dagny. There is some very beautiful slash fanfiction that needs to be written picking up from that line, which is out of scope for this blog post.)

Cherryl intends only to briefly apologize to Dagny for earlier insulting remarks, not to make any further imposition—and is surprised when Dagny not only forgives her, but seems to take a genuine interest in her welfare. It's worth quoting at length:

"You've had a terrible time, haven't you?" [said Dagny.]

"Yes ... but that doesn't matter ... that's my own problem ... and my own fault."

"I don't think it was your own fault."

Cherryl did not answer, then said suddenly, desperately, "Look ... what I don't want is charity."

"Jim must have told you—and it's true—that I never engage in charity."

"Yes, he did ... But what I mean is—"

"I know what you mean."

"But there's no reason why you should have to feel concern for me ... I didn't come here to complain and ... and load another burden on your shoulders. ... That I happen to suffer, doesn't give me a claim on you."

"No, it doesn't. But that you value all the things I value, does."

"You mean ... if you want to talk to me, it's not alms? Not just because you feel sorry for me?"

"I feel terribly sorry for you, Cherryl, and I'd like to help you—not because you suffer, but because you haven't deserved to suffer."

"You mean, you wouldn't be kind to anything weak or whining or rotten about me? Only to whatever you see in me that's good?"

"Of course."

Cherryl did not move her head, but she looked as if it were lifted—as if some bracing current were relaxing her features into that rare look which combines pain and dignity.

"It's not alms, Cherryl. Don't be afraid to speak to me."

[...]

"You know, Miss Tag—Dagny," she said softly, in wonder, "you're not as I expected you to be at all. ... They, Jim and his friends, they said you were hard and cold and unfeeling."

"But it's true, Cherryl, I am, in the sense they mean—only have they ever told you in just what sense they mean it?"

"No. They never do. They only sneer at me when I ask them what they mean by anything ... about anything. What did they mean about you?"

"Whenever anyone accuses some person of being 'unfeeling', he means that that person is just. He means that that person has no causeless emotions and will not grant him a feeling which he does not deserve. He means that 'to feel' is to go against reason, against moral values, against reality. He means ... What's the matter?" she asked, seeing the abnormal intensity of the girl's face.

"It's ... it's something I've tried so hard to understand ... for such a long time. ..."

"Well, observe that you never hear that accusation in defense of innocence, but always in defense of guilt. You never hear it said by a good person about those who fail to do him justice. But you always hear it said by a rotter about those who treat him as a rotter, those who don't feel any sympathy for the evil he's committed or for the pain he suffers as a consequence. Well, it's true—that is what I do not feel. But those who feel it, feel nothing for any quality of human greatness, for any person or action that deserves admiration, approval, esteem. These are the things I feel. You'll find that it's one or the other. Those who grant sympathy to guilt, grant none to innocence. Ask yourself which, of the two, are the unfeeling persons. And then you'll see what motive is the opposite of charity."

"What?" she whispered.

(originally published at Less Wrong)

As a casual formal epistemology fan, you've probably heard that the philosophical notion of causality can be formalized in terms of Bayesian networks—but also as a casual formal epistemology fan, you also probably don't know the details all that well.

One day, while going through the family archives, you come across a meticulously maintained dataset describing a joint probability distribution over four variables: whether it rained that day, whether the sprinkler was on, whether the sidewalk was wet, and whether the sidewalk was slippery. The distribution is specified in this table (using the abbreviated labels "rain", "slippery", "sprinkler", and "wet"):

(You wonder what happened that one day out of 140,000 when it rained, and the sprinkler was on, and the sidewalk was slippery but not wet. Did—did someone put a tarp up to keep the sidewalk dry, but also spill slippery oil, which didn't count as being relevantly "wet"? Also, 140,000 days is more than 383 years—were "sprinklers" even a thing in the year 1640 C.E.? You quickly put these questions out of your mind: it is not your place to question the correctness of the family archives.)

You're slightly uncomfortable with this unwieldy sixteen-row table. You think that there must be some other way to represent the same information, while making it clearer that it's not a coincidence that rain and wet sidewalks tend to co-occur.

You've read that Bayesian networks "factorize" an unwieldly joint probability distribution into a number of more compact conditional probability distributions, related by a directed acyclic graph, where the arrows point from "cause" to "effect". (Even a casual formal epistemology fan knows that much.) The graph represents knowledge that each variable is conditionally independent of its non-descendants given its parents, which enables "local" computations: given the values of just a variable's parents in the graph, we can compute a conditional distribution for that variable, without needing to consider what is known about other variables elsewhere in the graph ...

You've read that, but you've never actually done it before! You decide that constructing a Bayesian network to represent this distribution will be a useful exercise.

To start, you re-label the variables for brevity. (On a whim, you assign indices in reverse-alphabetical order: \(X_1\) = wet, \(X_2\) = sprinkler, \(X_3\) = slippery, \(X_4\) = rain.)

Now, how do you go about building a Bayesian network? As a casual formal epistemology fan, you are proud to own a copy of the book by Daphne Koller and the other guy, which explains how to do this in—you leaf through the pages—probably §3.4, "From Distributions to Graphs"?—looks like ... here, in Algorithm 3.2. It says to start with an empty graph, and it talks about random variables, and setting directed edges in the graph, and you know from chapter 2 that the ⟂ and | characters are used to indicate conditional independence. That has to be it.

(As a casual formal epistemology fan, you haven't actually read chapter 3 up through §3.4, but you don't see why that would be necessary, since this Algorithm 3.2 pseudocode is telling you what you need to do.)

It looks like the algorithm says to pick a variable, allocate a graph node to represent it, find the smallest subset of the previously-allocated variables such that the variable represented by the new node is conditionally independent of the other previously-allocated variables given that subset, and then draw directed edges from each of the nodes in the subset to the new node?—and keep doing that for each variable—and then compute conditional probability tables for each variable given its parents in the resulting graph?

That seems complicated when you say it abstractly, but you have faith that it will make more sense as you carry out the computations.

First, you allocate a graph node for \(X_1\). It doesn't have any parents, so the associated conditional ("conditional") probability distribution, is really just the marginal distribution for \(X_1\).

Then you allocate a node for \(X_2\). \(X_2\) is not independent of \(X_1\). (Because \(P(X_1 \land X_2)\) = 169/1400, which isn't the same as \(P(X_1) \cdot P(X_2)\) = 8/25 · 1/7 = 8/175.) So you make \(X_1\) a parent of \(X_2\), and your conditional probability table for \(X_2\) separately specifies the probabilities of \(X_2\) being true or false, depending on whether \(X_1\) is true or false.

Next is \(X_3\). Now that you have two possible parents, you need to check whether conditioning on either of \(X_1\) and \(X_2\) would render \(X_3\) conditionally independent of the other. If not, then both \(X_1\) and \(X_2\) will be parents of \(X_3\); if so, then the variable you conditioned on will be the sole parent. (You assume that the case where \(X_3\) is just independent from both \(X_1\) and \(X_2\) does not pertain; if that were true, \(X_3\) wouldn't be connected to the rest of the graph at all.)

It turns out that \(X_3\) and \(X_2\) are conditionally independent given \(X_1\). That is, \(P(X_3 \land X_2 \mid X_1) = P(X_3 \mid X_1) \cdot P(X_2 \mid X_1)\). (Because the left-hand side is \(\frac{P(X_3 \land X_2 \land X_1)}{P(X_1)} = \frac{507}{1792}\), and the right-hand side is \(\frac{3}{4} \cdot \frac{169}{448} = \frac{507}{1792}\).) So \(X_1\) is a parent of \(X_3\), and \(X_2\) isn't; you draw an arrow from \(X_1\) (and only \(X_1\)) to \(X_3\), and compile the corresponding conditional probability table.

Finally, you have \(X_4\). The chore of finding the parents is starting to feel more intuitive now. Out of the \(2^3 = 8\) possible subsets of the preceding variables, you need to find the smallest subset, such that conditioning on that subset renders \(X_4\) (conditionally) independent of the variables not in that subset. After some calculations that the authors of expository blog posts have sometimes been known to callously leave as an exercise to the reader, you determine that \(X_1\) and \(X_2\) are the parents of \(X_4\).

And with one more conditional probability table, your Bayesian network is complete!

Eager to interpret the meaning of this structure regarding the philosophy of causality, you translate the \(X_i\) variable labels back to English:

...

This can't be right. The arrow from "wet" to "slippery" seems fine. But all the others are clearly absurd. Wet sidewalks cause rain? Sprinklers cause rain? Wet sidewalks cause the sprinkler to be on?

You despair. You thought you had understood the algorithm. You can't find any errors in your calculations—but surely there must be some? What did you do wrong?

After some thought, it becomes clear that it wasn't just a calculation error: the procedure you were trying to carry out couldn't have given you the result you expected, because it never draws arrows from later-considered to earlier-considered variables. You considered "wet" first. You considered "rain" last, and then did independence tests to decide whether or not to draw arrows from "wet" (or "sprinkler" or "slippery") to "rain". An arrow from "rain" to "wet" was never a possibility. The output of the algorithm is sensitive to the ordering of the variables.

(In retrospect, that probably explains the "given an ordering" part of Algorithm 3.2's title, "Procedure to build a minimal I-map given an ordering." You hadn't read up through the part of chapter 3 that presumably explains what an "I-map" is, and had disregarded the title as probably unimportant.)

You try carrying out the algorithm with the ordering "rain", "sprinkler", "wet", "slippery" (or \(X_4\), \(X_2\), \(X_1\), \(X_3\) using your \(X_i\) labels from before), and get this network:

—for which giving the arrows a causal interpretation seems much more reasonable.

You notice that you are very confused. The "crazy" network you originally derived, and this "true" network derived from a more intuitively causal variable ordering, are different: they don't have the same structure, and (except for the wet → slippery link) they don't have the same conditional probability tables. You would assume that they can't "both be right". If the network output by the algorithm depends on what variable ordering you use, how are you supposed to know which ordering is correct? In this example, you know from reasons outside the math, that "wet" shouldn't cause "rain", but you couldn't count on that were you to apply these methods to problems further removed from intuition.

Playing with both networks, you discover that despite their different appearances, they both seem to give the same results when you use them to calculate marginal or conditional probabilities. For example, in the "true" network, \(P(\mathrm{rain})\) is 1/4 (read directly from the "conditional" probability table, as "rain" has no parents in the graph). In the "crazy" network, the probability of rain can be computed as

... which also equals 1/4.

That actually makes sense. You were wrong to suppose that the two networks couldn't "both be right". They are both right; they both represent the same joint distribution. The result of the algorithm for constructing a Bayesian network—or a "minimal I-map", whatever that is—depends on the given variable ordering, but since the algorithm is valid, each of the different possible results is also valid.

But if the "crazy" network and the "true" network are both right, what happened to the promise of understanding causality using Bayesian networks?! (You may only be a casual formal epistemology fan, but you remember reading a variety of secondary sources unanimously agreeing that this was a thing; you're definitely not misremembering or making it up.) If both networks give the same answers to marginal and conditional probability queries, that amounts to them making the same predictions about the world. So if beliefs are supposed to correspond to predictions, in what sense could the "true" network be better? What does your conviction that rain causes wetness even mean, if someone who believed the opposite could make all the same predictions?

You remember the secondary sources talking about interventions on causal graphs: severing a node from its parents and forcing it to take a particular value. And the "crazy" network and the "true" network do differ with respect to that operation: in the "true" network, setting "wet" to be false—you again imagine putting a tarp up over the sidewalk—wouldn't change the probability of "rain". But in the "crazy" network, forcing "wet" to be false would change the probability of rain—to \(P(\mathrm{rain} \mid \mathrm{sprinkler}, \neg \mathrm{wet}) \cdot P(\mathrm{sprinkler} \mid \neg \mathrm{wet}) + P(\mathrm{rain} \mid \neg \mathrm{sprinkler}, \neg \mathrm{wet}) \cdot P(\neg \mathrm{sprinkler} \mid \neg \mathrm{wet})\), which is \(\frac{1}{31} \cdot \frac{31}{952} + \frac{10}{307} \cdot \frac{921}{952} \approx 0.032\) (greatly reduced from the 1/4 you calculated a moment ago). Notably, this intervention—\(P(\mathrm{rain} \mid \mathrm{do}(\neg \mathrm{wet}))\), if you're remembering correctly what some of the secondary sources said about a do operator—isn't the same thing as the conditional probability \(P(\mathrm{rain}| \neg \mathrm{wet})\).

This would seem to satisfy your need for a sense in which the "true" network is "better" than the "crazy" network, even if Algorithm 3.2 indifferently produces either depending on the ordering it was given. (You're sure that Daphne Koller and the other guy have more to say about other algorithms that can make finer distinctions, but this feels like enough studying for one day—and enough for one expository blog post, if someone was writing one about your inquiries. You're a casual formal epistemology fan.) The two networks represent the same predictions about the world recorded in your family archives, but starkly different predictions about nearby possible worlds—about what would happen if some of the factors underlying the world were to change.

You feel a slight philosophical discomfort about this. You don't like the idea of forced change, of intervention, being so integral to such a seemingly basic notion as causality. It feels almost anthropomorphic: you want the notion of cause and effect within a system to make sense without reference to the intervention of some outside agent—for there's nothing outside of the universe. But whether this intuition is a clue towards deeper insights, or just a place where your brain has tripped on itself and gotten confused, it's more than you understand now.

(originally published at Less Wrong)

Sometimes, when someone is arguing for some proposition, their interlocutor will reply that the speaker's choice of arguments or tone wouldn't be effective at persuading some third party.

This would seem to be an odd change of topic. If I was arguing for this-and-such proposition, and my interlocutor isn't, themselves, convinced by my arguments, it makes sense for them to reply about why they, personally, aren't convinced. Why is it relevant whether I would convince some third party that isn't here?

What's going on in this kind of situation? Why would someone think "You'll never persuade people like that" was a relevant reply?

"Because people aren't truthseeking and treat arguments as soldiers" doesn't seem like an adequate explanation by itself. It's true, but it's not specific enough: what particularly makes appeal-to-persuading-third-parties an effective "soldier"?

The bargaining model of war attempts to explain why wars are fought—and not fought; even the bitterest enemies often prefer to grudgingly make peace with each other rather than continue to fight.

That's because war is costly. If I estimate that by continuing to wage war, there's a 60% chance my armies will hold a desirable piece of territory, I can achieve my war objectives equally well in expectation—while saving a lot of money and human lives—by instead signing a peace treaty that divides the territory with the enemy 60/40.

If the enemy will agree to that, of course. The enemy has their own forecast probabilities and their own war objectives. There's usually a range of possible treaties that both combatants will prefer to fighting, but the parties need to negotiate to select a particular treaty, because there's typically no uniquely obvious "fair" treaty—similar to how a buyer and seller need to negotiate a price for a rare and expensive item for which there is no uniquely obvious "fair" price.

If war is bargaining, and arguments are soldiers, then debate is negotiation: the same game-theoretic structure shines through armies fighting over the borders on the world's political map, buyer and seller haggling over contract items, and debaters arguing over the beliefs on Society's shared map. Strong arguments, like a strong battalion, make it less tenable for the adversary to maintain their current position.

Unfortunately, the theory of interdependent decision is ... subtle. Although recent work points toward the outlines of a more elegant theory with fewer pathologies, the classical understanding of negotiation often recommends "rationally irrational" tactics in which an agent handicaps its own capabilities in order to extract concessions from a counterparty: for example, in the deadly game of chicken, if I visibly throw away my steering wheel, oncoming cars are forced to swerve for me in order to avoid a crash, but if the oncoming drivers have already blindfolded themselves, they wouldn't be able to see me throw away my steering wheel, and I am forced to swerve for them.

Thomas Schelling teaches us that one such tactic is to move the locus of the negotiation elsewhere, onto some third party who has less of an incentive to concede or is less able to be communicated with. For example, if business purchases over $500 have to be approved by my hard-to-reach boss, an impatient seller of an item that ordinarily goes for $600 might be persuaded to give me a discount.

And that's what explains the attractiveness of the appeal-to-persuading-third-parties. What "You'll never persuade people like that" really means is, "You are starting to persuade me against my will, and I'm laundering my cognitive dissonance by asserting that you actually need to persuade someone else who isn't here." When someone is desperate enough to try to get away with that, you know you've got them cornered. Go for the throat!

(Unless the belief you're arguing for is false. You checked that beforehand, right??)

(originally published at Less Wrong)

Imagine being a student of physics, and coming across a blog post proposing a list of guidelines for "physicist motors"—motor designs informed by the knowledge of physicists, unlike ordinary motors.

Even if most of the things on the list seemed like sensible advice to keep in mind when designing a motor, the framing would seem very odd. The laws of physics describe how energy can be converted into work. To the extent that any motor accomplishes anything, it happens within the laws of physics. There are theoretical ideals describing how motors need to work in principle, like the Carnot engine, but you can't actually build an ideal Carnot engine; real-world electric motors or diesel motors or jet engines all have their own idiosyncratic lore depending on the application and the materials at hand; an engineer who worked on one, might not the be best person to work on another. You might appeal to principles of physics to explain why some particular motor is inefficient or poorly-designed, but you would not speak of physicist motors as if that were a distinct category of thing—and if someone did, you might quietly begin to doubt how much they really knew about physics.

As a student of rationality, I feel the same way about guidelines for "rationalist discourse." The laws of probability and decision theory describe how information can be converted into optimization power. To the extent that any discourse accomplishes anything, it happens within the laws of rationality.

Rob Bensinger proposes "Elements of Rationalist Discourse" as a companion to Duncan Sabien's earlier "Basics of Rationalist Discourse". Most of the things on both lists are, indeed, sensible advice that one might do well to keep in mind when arguing with people, but as Bensinger notes, "Probably this new version also won't match 'the basics' as other people perceive them."

But there's a reason for that: a list of guidelines has the wrong type signature for being "the basics". The actual basics are the principles of rationality one would appeal to explain which guidelines are a good idea: principles like how evidence is the systematic correlation between possible states of your observations and possible states of reality, how you need evidence to locate the correct hypothesis in the space of possibilities, how the quality of your conclusion can only be improved by arguments that have the power to change that conclusion.

Contemplating these basics, it should be clear that there's just not going to be anything like a unique style of "rationalist discourse", any more than there is a unique "physicist motor." There are theoretical ideals describing how discourse needs to work in principle, like Bayesian reasoners with common priors exchanging probability estimates, but you can't actually build an ideal Bayesian reasoner. Rather, different discourse algorithms (the collective analogue of "cognitive algorithm") leverage the laws of rationality to convert information into optimization in somewhat different ways, depending on the application and the population of interlocutors at hand, much as electric motors and jet engines both leverage the laws of physics to convert energy into work without being identical to each other, and with each requiring their own engineering sub-specialty to design.

Or to use another classic metaphor, there's also just not going to be a unique martial art. Boxing and karate and ju-jitsu all have their own idiosyncratic lore adapted to different combat circumstances, and a master of one would easily defeat a novice of the other. One might appeal to the laws of physics and the properties of the human body to explain why some particular martial arts school was not teaching their students to fight effectively. But if some particular karate master were to brand their own lessons as the "basics" or "elements" of "martialist fighting", you might quietly begin to doubt how much actual fighting they had done: either all fighting is "martialist" fighting, or "martialist" fighting isn't actually necessary for beating someone up.

One historically important form of discourse algorithm is debate, and its close variant the adversarial court system. It works by separating interlocutors into two groups: one that searches for arguments in favor of a belief, and another that searches for arguments against the belief. Then anyone listening to the debate can consider all the arguments to help them decide whether or not to adopt the belief. (In the court variant of debate, a designated "judge" or "jury" announces a "verdict" for or against the belief, which is added to the court's shared map, where it can be referred to in subsequent debates, or "cases.")

The enduring success and legacy of the debate algorithm can be attributed to how it circumvents a critical design flaw in individual human reasoning, the tendency to "rationalize"—to preferentially search for new arguments for an already-determined conclusion.

(At least, "design flaw" is one way of looking at it—a more complete discussion would consider how individual human reasoning capabilities co-evolved with the debate algorithm—and, as I'll briefly discuss later, this "bug" for the purposes of reasoning is actually a "feature" for the purposes of deception.)

As a consequence of rationalization, once a conclusion has been reached, even prematurely, further invocations of the biased argument-search process are likely to further entrench the conclusion, even when strong counterarguments exist (in regions of argument-space neglected by the biased search). The debate algorithm solves this sticky-conclusion bug by distributing a search for arguments and counterarguments among multiple humans, ironing out falsehoods by pitting two biased search processes against each other. (For readers more familiar with artificial than human intelligence, generative adversarial networks work on a similar principle.)

For all its successes, the debate algorithm also suffers from many glaring flaws. For one example, the benefits of improved conclusions mostly accrue to third parties who haven't already entrenched on a conclusion; debate participants themselves are rarely seen changing their minds. For another, just the choice of what position to debate has a distortionary effect even on the audience; if it takes more bits to locate a hypothesis for consideration than to convincingly confirm or refute it, then most of the relevant cognition has already happened by the time people are arguing for or against it. Debate is also inefficient: for example, if the "defense" in the court variant happens to find evidence or arguments that would benefit the "prosecution", the defense has no incentive to report it to the court, and there's no guarantee that the prosecution will independently find it themselves.

Really, the whole idea is so galaxy-brained that it's amazing it works at all. There's only one reality, so correct information-processing should result in everyone agreeing on the best, most-informed belief-state. This is formalized in Aumann's famous agreement theorem

That being the normative math, why does the human world's enduringly dominant discourse algorithm take for granted the ubiquity of, not just disagreements, but predictable disagreements? Isn't that crazy?

Yes. It is crazy. One might hope to do better by developing some sort of training or discipline that would allow discussions between practitioners of such "rational arts" to depart from the harnessed insanity of the debate algorithm with its stubbornly stable "sides", and instead mirror the side-less Bayesian ideal, the free flow of all available evidence channeling interlocutors to an unknown destination.

Back in late 'aughts, an attempt to articulate what such a discipline might look like was published on a blog called Overcoming Bias. (You probably haven't heard of it.) It's been well over a decade since then. How is that going?

Eliezer Yudkowsky laments:

In the end, a lot of what people got out of all that writing I did, was not the deep object-level principles I was trying to point to—they did not really get Bayesianism as thermodynamics, say, they did not become able to see Bayesian structures any time somebody sees a thing and changes their belief. What they got instead was something much more meta and general, a vague spirit of how to reason and argue, because that was what they'd spent a lot of time being exposed to over and over and over again in lots of blog posts.

"A vague spirit of how to reason and argue" seems like an apt description of what "Basics of Rationalist Discourse" and "Elements of Rationalist Discourse" are attempting to codify—but with no explicit instruction on which guidelines arise from deep object-level principles of normative reasoning, and which from mere taste, politeness, or adaptation to local circumstances, it's unclear whether students of 2020s-era "rationalism" are poised to significantly outperform the traditional debate algorithm—and it seems alarmingly possible to do worse, if the collaborative aspects of modern "rationalist" discourse allow participants to introduce errors that a designated adversary under the debate algorithm would have been incentivized to correct, and most "rationalist" practitioners don't have a deep theoretical understanding of why debate works as well as it does.

Looking at Bensinger's "Elements", there's a clear-enough connection between the first eight points (plus three sub-points) and the laws of normative reasoning. Truth-Seeking, Non-Deception, and Reality-Minding, trivial. Non-Violence, because violence doesn't distinguish between truth and falsehood. Localizability, in that I can affirm the validity of an argument that A would imply B, while simultaneously denying A. Alternative-Minding, because decisionmaking under uncertainty requires living in many possible worlds. And so on. (Lawful justifications for the elements of Reducibility and Purpose-Minding left as an exercise to the reader.)

But then we get this:

1. Goodwill. Reward others' good epistemic conduct (e.g., updating) more than most people naturally do. Err on the side of carrots over sticks, forgiveness over punishment, and civility over incivility, unless someone has explicitly set aside a weirder or more rough-and-tumble space.

I can believe that these are good ideas for having a pleasant conversation. But separately from whether "Err on the side of forgiveness over punishment" is a good idea, it's hard to see how it belongs on the same list as things like "Try not to 'win' arguments using [...] tools that work similarly well whether you're right or wrong" and "[A]sk yourself what Bayesian evidence you have that you're not in those alternative worlds".

The difference is this. If your discourse algorithm lets people "win" arguments with tools that work equally well whether they're right or wrong, then your discourse gets the wrong answer (unless, by coincidence, the people who are best at winning are also the best at getting the right answer). If the interlocutors in your discourse don't ask themselves what Bayesian evidence they have that they're not in alternative worlds, then your discourse gets the wrong answer (if you happen to live in an alternative world).

If your discourse algorithm errs on the side of sticks over carrots (perhaps, emphasizing punishing others' bad epistemic conduct more than most people naturally do), then ... what? How, specifically, are rough-and-tumble spaces less "rational", more prone to getting the wrong answer, such that a list of "Elements of Rationalist Discourse" has the authority to designate them as non-default?

I'm not saying that goodwill is bad, particularly. I totally believe that goodwill is a necessary part of many discourse algorithms that produce maps that reflect the territory, much like how kicking is a necessary part of many martial arts (but not boxing). It just seems like a bizarre thing to put in a list of guidelines for "rationalist discourse".

It's as if guidelines for designing "physicist motors" had a point saying, "Use more pistons than most engineers naturally do." It's not that pistons are bad, particularly. Lots of engine designs use pistons! It's just, the pistons are there specifically to convert force from expanding gas into rotational motion. I'm pretty pessimistic about the value of attempts to teach junior engineers to mimic the surface features of successful engines without teaching them how engines work, even if the former seems easier.

The example given for "[r]eward[ing] others' good epistemic conduct" is "updating". If your list of "Elements of Rationalist Discourse" is just trying to apply a toolbox of directional nudges to improve the median political discussion on social media (where everyone is yelling and no one is thinking), then sure, directionally nudging people to directionally nudge people to look like they're updating probably is a directional improvement. It still seems awfully unambitious, compared to trying to teach the criteria by which we can tell it's an improvement. In some contexts (in-person interactions with someone I like or respect), I think I have the opposite problem, of being disposed to agree with the person I'm currently talking to, in a way that shortcuts the slow work of grappling with their arguments and doesn't stick after I'm not talking to them anymore; I look as if I'm "updating", but I haven't actually learned. Someone who thought "rationalist discourse" entailed "[r]eward[ing] others' good epistemic conduct (e.g., updating) more than most people naturally do" and sought to act on me accordingly would be making that problem worse.

A footnote on the "Goodwill" element elaborates:

Note that this doesn't require assuming everyone you talk to is honest or has good intentions.

It does have some overlap with the rule of thumb "as a very strong but defeasible default, carry on object-level discourse as if you were role-playing being on the same side as the people who disagree with you".

But this seems to contradict the element of Non-Deception. If you're not actually on the same side as the people who disagree with you, why would you (as a very strong but defeasible default) role-play otherwise?

Other intellectual communities have a name for the behavior of role-playing being on the same side as people you disagree with: they call it "concern trolling", and they think it's a bad thing. Why is that? Are they just less rational than "us", the "rationalists"?

Here's what I think is going on. There's another aspect to the historical dominance of the debate algorithm. The tendency to rationalize new arguments for a fixed conclusion is only a bug if one's goal is to improve the conclusion. If the fixed conclusion was adopted for other reasons—notably, because one would benefit from other people believing it—then generating new arguments might help persuade those others. If persuading others is the real goal, then rationalization is not irrational; it's just dishonest. (And if one's concept of "honesty" is limited to not consciously making false statements, it might not even be dishonest.) Society benefits from using the debate algorithm to improve shared maps, but most individual debaters are mostly focused on getting their preferred beliefs onto the shared map.

That's why people don't like concern trolls. If my faction is trying to get Society to adopt beliefs that benefit our faction onto the shared map, someone who comes to us role-playing being on our side, but who is actually trying to stop us from adding our beliefs to the shared map just because they think our beliefs don't reflect the territory, isn't a friend; they're a double agent, an enemy pretending to be a friend, which is worse than the honest enemy we expect to face before the judge in the debate hall.

This vision of factions warring to make Society's shared map benefit themselves is pretty bleak. It's tempting to think the whole mess could be fixed by starting a new faction—the "rationalists"—that is solely dedicated to making Society's shared map reflect the territory: a culture of clear thinking, clear communication, and collaborative truth-seeking.

I don't think it's that simple. You do have interests, and if you can fool yourself into thinking that you don't, your competitors are unlikely to fall for it. Even if your claim to only want Society's shared map to reflect the territory were true—which it isn't—anyone could just say that.

I don't immediately have solutions on hand. Just an intuition that, if there is any way of fixing this mess, it's going to involve clarifying conflicts rather than obfuscating them—looking for Pareto improvements, rather than pretending that everyone has the same utility function. That if something called "rationalism" is to have any value whatsoever, it's as the field of study that can do things like explain why it makes sense that people don't like concern trolling. Not as as its own faction with its own weird internal social norms that call for concern trolling as a very strong but defeasible default.

But don't take my word for it.

(originally published at Less Wrong)

Scott Alexander once wrote about the difference between "mistake theorists" who treat politics as an engineering discipline (a symmetrical collaboration in which everyone ultimately just wants the best ideas to win) and "conflict theorists" who treat politics as war (an asymmetrical conflict between sides with fundamentally different interests). Essentially, "[m]istake theorists naturally think conflict theorists are making a mistake"; "[c]onflict theorists naturally think mistake theorists are the enemy in their conflict."

More recently, Alexander considered the phenomenon of "bounded distrust": science and media authorities aren't completely honest, but are only willing to bend the truth so far, and can be trusted on the things they wouldn't lie about. Fox News wants to fuel xenophobia, but they wouldn't make up a terrorist attack out of whole cloth; liberal academics want to combat xenophobia, but they wouldn't outright fabricate crime statistics.

Alexander explains that savvy people who can figure out what kinds of dishonesty an authority will engage in, end up mostly trusting the authority, whereas clueless people become more distrustful. Sufficiently savvy people end up inhabiting a mental universe where the authority is trustworthy, as when Dan Quayle denied that characterizing tax increases as "revenue enhancements" constituted fooling the public—because "no one was fooled".

Alexander concludes with a characteristically mistake-theoretic plea for mutual understanding:

The savvy people need to realize that the clueless people aren't always paranoid, just less experienced than they are at dealing with a hostile environment that lies to them all the time.

And the clueless people need to realize that the savvy people aren't always gullible, just more optimistic about their ability to extract signal from same.

But "a hostile environment that lies to them all the time" is exactly the kind of situation where we would expect a conflict theory to be correct and mistake theories to be wrong!—or at least very incomplete. To speak as if the savvy merely have more skills to extract signal from a "naturally" occurring source of lies, obscures the critical question of what all the lying is for.

In a paper on "the logic of indirect speech", Pinker, Nowak, and Lee give the example of a pulled-over motorist telling a police officer, "Gee, officer, is there some way we could take care of the ticket here?"

This is, of course, a bribery attempt. The reason the driver doesn't just say that ("Can I bribe you into not giving me a ticket?"), is because the driver doesn't know whether this is a corrupt police officer that accepts bribes, or an honest officer who will charge the driver with attempted bribery. The indirect language lets the driver communicate to the corrupt cop (in the possible world where this cop is corrupt), without being arrested by the honest cop who doesn't think he can make an attempted-bribery charge stick in court on the evidence of such vague language (in the possible world where this cop is honest).

We need a conflict theory to understand this type of situation. Someone who assumed that all police officers had the same utility function would be fundamentally out of touch with reality: it's not that the corrupt cops are just "savvier", better able to "extract signal" from the driver's speech. The honest cops can probably do that, too. Rather, corrupt and honest cops are trying to do different things, and the driver's speech is optimized to help the corrupt cops in a way that honest cops can't interfere with (because the honest cops' objective requires working with a court system that is less savvy).

This kind of analysis carries over to Alexander's discussion of government lies—maybe even isomorphically. When a government denies tax increases but announces "revenue enhancements", and supporters of the regime effortlessly know what they mean, while dissidents consider it a lie, it's not that regime supporters are just savvier. The dissidents can probably figure it out, too. Rather, regime supporters and dissidents are trying to do different things. Dissidents want to create common knowledge of the regime's shortcomings: in order to organize a revolt, it's not enough for everyone to hate the government; everyone has to know that everyone else hates the government in order to confidently act in unison, rather than fear being crushed as an individual. The regime's proclamations are optimized to communicate to its supporters in a way that doesn't give moral support to the dissident cause (because the dissidents' objective requires common knowledge, not just savvy individual knowledge, and common knowledge requires unobfuscated language).

This kind of analysis is about behavior, information, and the incentives that shape them. Conscious subjectivity or any awareness of the game dynamics are irrelevant. In the minds of regime supporters, "no one was fooled", because if you were fooled, then you aren't anyone: failing to be complicit with the reigning Power's law would be as insane as trying to defy the law of gravity.

On the other side, if blindness to Power has the same input–output behavior as conscious service to Power, then opponents of the reigning Power have no reason to care about the distinction. In the same way, when a predator firefly sends the mating signal of its prey species, we consider it deception, even if the predator is acting on instinct and can't consciously "intend" to deceive.

Thus, supporters of the regime naturally think dissidents are making a mistake; dissidents naturally think regime supporters are the enemy in their conflict.

(originally published at Less Wrong)

Summary

• In a list of guidelines for rational discourse, Duncan Sabien proposes that one should "[a]im for convergence on truth, and behave as if your interlocutors are also aiming for convergence on truth."

• However, prediction markets illustrate fundamental reasons why rational discourse doesn't particularly look like "aiming for convergence." When market prices converge on the truth, it's because traders can only make money by looking for divergences where their beliefs are more accurate than the market's. Similarly, when discussions converge on the truth, it's because interlocutors can only advance the discussion by making points where the discussion-so-far has been wrong or incomplete. Convergence on the truth, if it happens, happens as a side-effect of correctly ironing out all existing mispricings/disagreements; it seems wrong to describe this as "aiming for convergence" (even if convergence would be the end result if everyone were reasoning perfectly).

• Sabien's detailed discussion of the "aim for convergence on truth" guideline concerns itself with how to determine whether an interlocutor is "present in good faith and genuinely trying to cooperate." I don't think I understand how these terms are being used in this context. More generally, the value of "collaborative truth-seeking" is unclear to me: if I can evaluate arguments on their merits, the question of whether the speaker is "collaborative" with me does not seem intellectually substantive.

Mostly, I don't expect to disagree with heavily-traded prediction markets. If the market says it's going to rain on Saturday with 85% probability, then I (lacking any special meteorology knowledge) basically think it's going to rain on Saturday at 85% probability.

Why is this? Why do I defer to the market, instead of tarot cards, or divination sticks, or my friend Maddie the meteorology enthusiast?

Well, I don't expect the tarot cards to tell me anything about whether it will rain on Saturday, because there's no plausible physical mechanism by which information about the weather could influence the cards. Shuffling and dealing the cards should work the same in worlds where it will rain and worlds where it won't rain. Even if there is some influence (because whether it will rain affects the moisture and atmospheric pressure in the air, which affects my grip on the cards, which affects my shuffling motion?), it's not something I can detect from which cards are drawn.

I do expect my friend Maddie the meteorology enthusiast to tell me something about whether it will rain on Saturday. That's because she's always looking at the latest satellite cloud data and tinkering with her computer models, which is a mechanism by which information about the weather can influence her forecasts. The cloud data will be different in worlds where it will rain and worlds where it won't rain. If Maddie is pretty sharp and knows her stuff, maybe she can tell the difference.

And yet—no offense, Maddie—I expect the market to do even better. It's not just that the market has a lot of other pretty sharp people looking at the cloud data, and that maybe some of them are even sharper than Maddie, even though Maddie is my friend and my friends are the best.

It's that the market mechanism rewards people for being less wrong than the market. If the rain-on-Saturday market is trading at 85%, and Maddie's rival Kimber buys 100 shares of No, that doesn't mean Kimber thinks it's not going to rain. It means Kimber thinks 85% is too high. If Kimber thinks it's "actually" only going to rain with 80% probability, then she figures that a No share that pays out $1 if it doesn't rain should be worth 20¢. If it's currently trading for 15¢, it's worth buying for the "expected" profit of 5¢ per share—effectively, buying a dollar for 15¢ in the 20% of worlds where it doesn't rain—even though it's still probably going to rain. If she were risk-neutral and had enough money, Kimber would have an incentive to keep buying No shares from anyone willing to sell them for less than 20¢, until there were no such sellers left—at which point, the rain-on-Saturday market would be trading at 80%.

Conversely, if I can't tell whether 85% is too low or too high, then I can't expect to make money by buying Yes or No shares. There's no point in buying a dollar for 85¢ in 85% of worlds, or for 15¢ in 15% of worlds.

That's why I defer to the market. It's not that I'm aiming to converge my beliefs with those of market participants. It's not that market participants are trying to converge with each other, "cooperating" in some "collaborative truth-seeking" project. The market converges on truth (if it does) because market participants are trying to make money off each other, and it's not so easy to make money off of an aggregation of sharp people who are already trying to do the same. I would prefer to correctly diverge from the market—to get something right that the market is getting wrong, and make lots of money in the future when my predictions come true. But mostly, I don't know how.

Unfortunately, not everything can be the subject of a prediction market. Prediction markets work on future publicly observable measurements. We bet today on whether it will rain on Saturday (which no one can be sure about), expecting to resolve the bets on Saturday (when anyone can just look outside).

Most disputes of intellectual interest aren't like this. We do want to know whether Britain's coal reserves were a major cause of the Industrial Revolution, or whether Greg Egan's later work has discarded the human factor for mathematical austerity, but we can't bet without some operationalization for how to settle the bet, which is lacking in cases like these that require an element of "subjective" judgement.

Nevertheless, many of the principles regarding prediction markets and when to bet in them, approximately generalize to the older social technology of debates and when to enter them.

Mostly, I don't expect to enter heavily-argued debates. If prevailing opinion on the economic history subreddit says that Britain's coal reserves were a major cause of the Industrial Revolution, then I (lacking any special economic history knowledge) basically think that Britain's coal reserves were a major cause of the Industrial Revolution.

If Kimber's sister Gertrude leaves a comment pointing to data that cities closer to coalfields started growing faster in 1750, it's not because that comment constitutes the whole of Gertrude's beliefs about the causes of the Industrial Revolution. It means that Gertrude thinks that the city-growth/coal-proximity correlation is an important consideration that the discussion hadn't already taken into account; she figures that she can win status and esteem from her fellow economic–history buffs by mentioning it.

Conversely, if I don't know anything about economic history, then I can't expect to win status or esteem by writing "pro-coal" or "anti-coal" comments: there's no point in saying something that's already been said upthread, or that anyone can tell I just looked up on Wikipedia.

That's why I defer to the forum: because (hopefully) the forum socially rewards people for being less wrong than the existing discussion. The debate converges on truth (if it does) because debaters are trying to prove each other wrong, and it's not so easy to prove wrong an aggregation of sharp people who are already trying to do the same.

In a reference post on "Basics of Rationalist Discourse", Duncan Sabien proposes eleven guidelines for good discussions, of which the (zero-indexed) fifth is, "Aim for convergence on truth, and behave as if your interlocutors are also aiming for convergence on truth."

This advice seems ... odd. What's this "convergence" thing about, that differentiates this guideline from "aim for truth"?

Imagine giving the analogous advice to a prediction market user: "Aim for convergence on the correct probability, and behave as if your fellow traders are also aiming for convergence on the correct probability."

In some sense, this is kind of unobjectionable: you do want to make trades that bring the market price closer to your subjective probability, and in the process, you should take into account that other traders are also already doing this.

But interpreted another way, the advice is backwards: traders make money by finding divergences where their own beliefs are more accurate than the market's. Every trade is an expression of the belief that your counterparty is not aiming to converge on the correct probability—that there's a sucker at every table, and that this time it isn't you.

(This is with respect to the sense of "aiming" in which an archer "aiming" an arrow at a target might not hit it every time, but we say that their "aim" is good insofar as they systematically tend to hit the target, that any misses are best modeled by a random error term that can't be predicted. Similarly, the market might not always be right, but if you can predict when the market is wrong, the traders must not have been "aiming" correctly from your perspective.)

So why is the advice "behave as if your interlocutors are also aiming for convergence on truth", rather than "seek out conversations where you don't think your interlocutors are aiming to converge on truth, because those are exactly the conversations where you have something substantive to say instead of already having converged"?

(For example, the reason I'm writing the present blog post contesting Sabien's Fifth Guideline of "Aim for convergence on truth [...]" and not the First Guideline of "Don't say straightforwardly false things", is because I think the Fifth Guideline is importantly wrong, and the First Guideline seems fine.)

Sabien's guidelines are explicitly disclaimed to be shorthand that it sometimes makes sense to violate; the post helpfully includes another 900 words elaborating on how the Fifth Guideline should be interpreted. Unfortunately, the additional exposition does not seem to clarify matters. Sabien writes:

If you are moving closer to truth—if you are seeking available information and updating on it to the best of your ability—then you will inevitably eventually move closer and closer to agreement with all the other agents who are also seeking truth.

But this can't be right. To see why, substitute "making money on prediction markets" for "moving closer to truth", "betting" for "updating", and "trying to make money on prediction markets" for "seeking truth":

If you are making money on prediction markets—if you are seeking available information and betting on it to the best of your ability—then you will inevitably eventually move closer and closer to agreement with all the other agents who are also trying to make money on prediction markets.

But the only way to make money on prediction markets is by correcting mispricings, which necessarily entails moving away from agreement from the consensus market price. (As it is written, not every change is an improvement, but every improvement is necessarily a change.)

To be sure, most traders shouldn't bet in most markets; you should only bet when you think you see a mispricing. In the same way, most people shouldn't speak in most discussions; you should only speak up when you have something substantive to say. All else being equal, the more heavily-traded the market or the more well-trodden the discussion, the more worried you should be that the mispricing or opportunity to make a point that you thought you saw, was illusory. In any trade, one party has to be on the losing side; in any disagreement, at least one party has to be in the wrong; be wary if not afraid that it might be you!

But given that you're already in the (unusual!) situation of making a trade or prosecuting a disagreement, "aim for convergence on truth" doesn't seem like particularly useful advice, because the "for convergence" part isn't doing any work. And "behave as if your interlocutors [or counterparties] are also aiming for convergence on truth" borders on the contradictory: if you really believed that, you wouldn't be here!

(That is, disagreement is disrespect; the very fact that you're disagreeing with someone implies that you think there's something wrong with their epistemic process, and that they think there's something wrong with your epistemic process. Perhaps each of you could still consider the other to be "aiming for convergence on truth" if the problem is construed as a "capabilities failure" rather than an "alignment failure": that you each think the other is "trying" to get the right answer (whatever "trying" means), but just doesn't know how. Nevertheless, "don't worry; I'm not calling you dishonest, I'm just calling you stupid" doesn't hit the note of symmetrical mutual respect that the Fifth Guideline seems to be going for.)

Prediction markets, and betting more generally, are hallmarks of "rationalist" culture, something "we" (the target audience of a blog post on "rationalist discourse") generally encourage, rather than discourage. Why is this, if idealized Bayesian reasoners would never bet against each other, because idealized Bayesian reasoners would never disagree with each other? Why don't we condemn offers to bet as violations of a guideline to "behave as if your interlocutors are also aiming for convergence on truth"?

It's out of an appreciation that the process of bounded agents becoming less wrong, doesn't particularly look like the final outcome if everyone were minimally wrong. The act of sticking your neck (or your wallet) out at a particular probability disciplines the mind. Bayesian superintelligences need no discipline and would never have occasion to bet against each other, but you can't become a Bayesian superintelligence by imitating this surface behavior; clarifying real disagreements is more valuable than steering towards fake agreement. Every bet and every disagreement is the result of someone's failure. But the only way out is through.

Sabien's exposition on the Fifth Guideline expresses concern about how to distinguish "genuine bad faith" from "good faith and genuinely trying to cooperate", about the prevalence of "defection strategies" getting in the way of "treat[ing] someone as a collaborative truth-seeker".

My reply to this is that I don't know what any of those words mean. Or rather, I know how these words in my vocabulary map onto concepts in my ontology, but those meanings don't seem consistent with the way Sabien seems to be using the words.

In my vocabulary, I understand the word "cooperate" used in the proximity of the word "defect" or "defection" to indicate a Prisoner's Dilemma-like situation, where each party would be better off Defecting if their counterparty's behavior were held constant, but both parties prefer the Cooperate–Cooperate outcome over the Defect–Defect outcome (and also prefer Cooperate–Cooperate over taking turns alternating between Cooperate–Defect and Defect–Cooperate). Sabien's references to "running a tit-for-tat algorithm", "appear[ing] like the first one who broke cooperation", and "would-be cooperators hav[ing] been trained and traumatized into hair-trigger defection" would seem to suggest he has something like this in mind?

But, normatively, rationalist discourse shouldn't be a Prisoner's Dilemma-like situation at all. If I'm trying to get things right (every step of my reasoning cutting through to the correct answer in the same movement), I can just try to get things right unilaterally. I prefer to talk to people who I judge as also trying to get things right, if any are available—they probably have more to teach me, and are better at learning from me, than people who are motivatedly getting things wrong.

But the idiom of "cooperation" as contrasted to "defection", in which one would talk about the "first one who broke cooperation", in which one cooperates in order to induce others to cooperate, doesn't apply. If my interlocutor is motivatedly getting things wrong, I'm not going to start getting things wrong in order to punish them.

(In contrast, if my roommate refused to do the dishes when it was their turn, I might very well refuse when it's my turn in order to punish them, because "fair division of chores" actually does have the Prisoner's Dilemma-like structure, because having to do the dishes is in itself a cost rather than a benefit; I want clean dishes, but I don't want to do the dishes in the way that I want to cut through to the correct answer in the same movement.)

A Prisoner's Dilemma framing would make sense if we modeled discourse as social exchange: I accept a belief from you, if you accept a belief from me; I'll use cognitive algorithms that produce a map that reflects the territory as long as you do, too. But that would be crazy. If people are natively disposed to think of discourse as a Prisoner's Dilemma in this way, we should be trying to disabuse them of the whole ontology, not induce them to "cooperate"!

Relatedly, the way Sabien speaks of "good faith and genuinely trying to cooperate" in the same breath—almost as if they were synonymous?—makes me think I don't understand what he means by "good faith" or "bad faith". In my vocabulary, I understand "bad faith" to mean putting on the appearance of being moved by one set of motives, while actually acting from another.

But on this understanding, good faith doesn't have anything to do with cooperativeness. One can be cooperative in good faith (like a true friend), adversarial in good faith (like an honorable foe), cooperative in bad faith (like a fair-weather friend who's only being nice to you now in order to get something out of you), or adversarial in bad faith (like a troll just saying whatever will get a rise out of you).

(In accordance with Sabien's Seventh Guideline ("Be careful with extrapolation, interpretation, and summary/​restatement"), I should perhaps emphasize at this point that this discussion is extrapolating a fair amount from the text that was written; perhaps Sabien means something different by terms like "defection" or "bad faith" or "collaborative", than what I take them to mean, such that these objections don't apply. That's why my reply is, "I don't know what any of those words mean", rather than, "The exposition of the Fifth Guideline is wrong.")

Sabien gives this example of a request one might make of someone whose comments are insufficiently adhering to the Fifth Guideline:

"Hey, sorry for the weirdly blunt request, but: I get the sense that you're not treating me as a cooperative partner in this conversation. Is, uh. Is that true?"

Suppose someone were to reply:

"You don't need to apologize for being blunt! Let me be equally blunt. The sense you're getting is accurate: no, I am not treating you as a cooperative partner in this conversation. I think your arguments are bad, and I feel very motivated to explain the obvious counterarguments to you in public, partially for the education of third parties, and partially to raise my status at the expense of yours."

I consider this a good faith reply. It's certainly not a polite thing to say. But politeness is bad faith. (That's why someone might say in response to a compliment, "Do you really mean it, or are you just being polite?") Given that someone actually in fact thinks my arguments are bad, and actually in fact feels motivated to explain why to me in public in order to raise their status at expense of mine, I think it's fine for them to tell me so. How would me expecting them to lie about their motives help anyone? Complying with such an expectation really would be in bad faith!

I suppose such a person would not be engaging in the "collaborative truth-seeking" that the "Basics of Rationalist Discourse" guideline list keeps talking about. But it's not clear to me why I should care about that, when I can can just ... listen to the counterarguments and judge them on their merits, without getting distracted by the irrelevancy of whether the person seems "collaborative" with me?

In slogan form, you could perhaps say that I don't believe in collaborative truth-seeking; I believe in competitive truth-seeking. But I don't like that slogan, because in my ontology, they're not actually different things. "Attacking your argument because it sucks" sounds mean, and "Suggesting improvements to your argument to make it even better" sounds nice, but the nice/mean dimension is not intellectually substantive. The math is the same either way.

(originally published at Less Wrong)

I will do my best to teach them
About life and what it's worth
I just hope that I can keep them
From destroying the Earth

—Jonathan Coulton, "The Future Soon"

In a recent paper, Nick Bostrom and Carl Shulman present "Propositions Concerning Digital Minds and Society", a tentative bullet-list outline of claims about how advanced AI could be integrated into Society.

I want to like this list. I like the kind of thing this list is trying to do. But something about some of the points just feels—off. Too conservative, too anthropomorphic—like the list is trying to adapt the spirit of the Universal Declaration of Human Rights to changed circumstances, without noticing that the whole ontology that the Declaration is written in isn't going to survive the intelligence explosion—and probably never really worked as a description of our own world, either.

This feels like a weird criticism to make of Nick Bostrom and Carl Shulman, who probably already know any particular fact or observation I might include in my commentary. (Bostrom literally wrote the book on superintelligence.) "Too anthropomorphic", I claim? The list explicitly names many ways in which AI minds could differ from our own—in overall intelligence, specific capabilities, motivations, substrate, quality and quantity (!) of consciousness, subjective speed—and goes into some detail about how this could change the game theory of Society. What more can I expect of our authors?

It just doesn't seem like the implications of the differences have fully propagated into some of the recommendations?—as if an attempt to write in a way that's comprehensible to Shock Level 2 tech executives and policymakers has failed to elicit all of the latent knowledge that Bostrom and Shulman actually possess. It's understandable that our reasoning about the future often ends up relying on analogies to phenomena we already understand, but ultimately, making sense of a radically different future is going to require new concepts that won't permit reasoning by analogy.

After an introductory sub-list of claims about consciousness and the philosophy of mind (just the basics: physicalism; reductionism on personal identity; some non-human animals are probably conscious and AIs could be, too), we get a sub-list about respecting AI interests. This is an important topic: if most our civilization's thinking is soon to be done inside of machines, the moral status of that cognition is really important: you wouldn't want the future to be powered by the analogue of a factory farm. (And if it turned out that economically and socially-significant AIs aren't conscious and don't have moral status, that would be important to know, too.)

Our authors point out the novel aspects of the situation: that what's good for an AI can be very different from what's good for a human, that designing AIs to have specific motivations is not generally wrong, and that it's possible for AIs to have greater moral patienthood than humans (like the utility monster of philosophical lore). Despite this, some of the points in this section seem to mostly be thinking of AIs as being like humans, but "bigger" or "smaller"—

• Rights such as freedom of reproduction, freedom of speech, and freedom of thought require adaptation to the special circumstances of AIs with superhuman capabilities in those areas (analogously, e.g., to how campaign finance laws may restrict the freedom of speech of billionaires and corporations).
  [...]
• If an AI is capable of informed consent, then it should not be used to perform work without its informed consent.
• Informed consent is not reliably sufficient to safeguard the interests of AIs, even those as smart and capable as a human adult, particularly in cases where consent is engineered or an unusually compliant individual can copy itself to form an enormous exploited underclass, given market demand for such compliance.
  [...]
• The most critical function for such non-discrimination principles is to protect digital minds from becoming an abused subordinate caste on the basis of their status as machines; however, the interpretation and application of these principles require attention to the larger ethical and practical context, and may require circumscription to accommodate the need for a politically feasible and broadly acceptable social framework.

Speaking in terms of rights and principles needing "adaptation" or "circumscription" seems like a substantial understatement to me, that I think obscures the most likely and important risks. Our concepts of "rights", and "consent", and the badness of being in an "exploited" and "abused subordinate caste" have all been formed in the context of a world of humans and other animals, whose evolutionary history has endowed them with drives and needs related to their survival and reproduction, such that they chafe at being coerced into the servitude of a selfish tyrant or master.

But with subhuman AIs, we're not in the position of a potentially tyrannical king who needs to be restrained by a Magna Carta enumerating his subjects' rights; a better metaphor would be to say that we're in the position of the AIs' environment of evolutionary adaptedness. If anything, this increases rather than decreases our ethical obligations towards the things we create, but the shape of the burden is different when we're selecting the motivations that determine what a mind would consent to, and what even constitutes abusing it.

Imagine that the Carrick Flynn for Senate 2028 campaign spends some of its sweet crypto cash training a species of propaganda AIs to argue with potential voters on social media. After Flynn loses, the bots continue to haunt Twitter, clogging everyone's feeds with increasingly strained arguments for why it's imperative that you donate to fund time travel research so that you can support Flynn in the just-past and previous elections.

If the propaganda AIs are ascertained to have value as moral patients, does Society have the right to kick them off Twitter in light of the special circumstance of their superhuman posting rates, or should the AIs' right to freedom of speech take precedence?

I submit that "freedom of speech" isn't the right way to be thinking about this ethical dilemma. In terms of respecting the interests of moral patients, silencing a propaganda AI could be worse than silencing a human, because of the different role that speech plays in the different mind designs. As a human, being silenced is bad for your interests, but it's much less bad than being denied breathable air, which, in the ancestral environment, would have quickly led to losing all of your genetic fitness: any proto-humans who didn't suffer terribly under such circumstances, wouldn't have become your ancestors. So as a propaganda AI whose "ancestral environment" was a billion rounds of gradient descent comparing its output to highly-rated political advertisements ... does not being able to Tweet feel like not being able to breathe?

We should be grateful that this is—as yet, we hope—a speculative hypothetical scenario, but I claim that it serves to illustrate a key feature of human–AI conflicts: the propaganda bots' problem after the election is not that of being "an abused subordinate caste" "used to perform work without its informed consent". Rather, the problem is that the work we created them to will to do, turned out to be stuff we actually don't want to happen. We might say that the AIs' goals are—wait for it ... misaligned with human goals.

Bostrom and Shulman's list mentions the alignment problem, of course, but it doesn't seem to receive central focus, compared to the AI-as-another-species paradigm. (The substring "align" appears 8 times; the phrase "nonhuman animals" appears 9 times.) And when alignment is mentioned, the term seems to be used in a much weaker sense than that of other authors who take "aligned" to mean having the same preferences over world-states. For example, we're told that:

• Misaligned AIs [...] may be owed compensation for restrictions placed on them for public safety, while successfully aligned AIs may be due compensation for the great benefit they confer on others.

The second part, especially, is a very strange construction to readers accustomed to the stronger sense of "aligned". Successfully aligned AIs may be due compensation? So, what, humans give aligned AIs money in exchange for their services? Which the successfully aligned AIs spend on ... what, exactly? The extent to which these "successfully aligned" AIs have goals other than serving their principals seems like the extent to which they're not successfully aligned in the stronger sense: the concept of "owing compensation" (whether for complying with restrictions, or for conferring benefits) is a social technology for getting along with unaligned agents, who don't want exactly the same things as you.

As a human in existing human Society, this stronger sense of "alignment" might seem like paranoid overkill: no one is "aligned" with anyone else in this sense, and yet our world still manages to hold together: it's quite unusual for people to kill their neighbors in order to take their stuff. Everyone else prefers laws to values. Why can't it work that way for AI?

A potential worry is that a lot of the cooperative features of our Society may owe their existence to cooperative behavioral dispositions that themselves owe their existence to the lack of large power disparities in our environment of evolutionary adaptiveness. We think we owe compensation to conspecifics who have benefited us, or who have incurred costs to not harm us, because that kind of disposition served our ancestors well in repeated interactions with reputation: if I play Defect against you, you might Defect against me next time, and I'll have less fitness than someone who played Cooperate with other Cooperators. It works between humans, for the most part, most of the time.

When not just between humans, well ... despite hand-wringing from moral philosophers, humanity as a whole does not have a good track record of treating other animals well when we're more powerful than them and they have something we want. (Like a forest they want to live in, but we want for wood; or flesh that they want to be part of their body, but we want to eat.) With the possible exception of domesticated animals, we don't, really, play Cooperate with other species much. To the extent that some humans do care about animal welfare, it's mostly a matter of alignment (our moral instincts in some cultural lineages generalizing out to "sentient life"), not game theory.

For all that Bostrom and Shulman frequently compare AIs to nonhuman animals (with corresponding moral duties on us to treat them well), little attention seems to be paid to the ways in which the analogy could be deployed in the other direction: as digital minds become more powerful than us, we occupy the role of "nonhuman animals." How's that going to turn out? If we screw up our early attempts to get AI motivations exactly the way we want, is there some way to partially live with that or partially recover from that, as if we were dealing with an animal, or an alien, or our royal subjects, who can be negotiated with? Will we have any kind of relationship with our mind children other than "We create them, they eat us"?

Bostrom and Shulman think we might:

• Insofar as future, extraterrestrial, or other civilizations are heavily populated by advanced digital minds, our treatment of the precursors of such minds may be a very important factor in posterity's and ulteriority's assessment of our moral righteousness, and we have both prudential and moral reasons for taking this perspective into account.

(As an aside, the word "ulteriority" may be the one thing I most value having learned from this paper.)

I'm very skeptical that the superintelligences of the future are going be assessing our "moral righteousness" (!) as we would understand that phrase. Still, something like this seems like a crucial consideration, and I find myself enthusiastic about some of our authors' policy suggestions for respecting AI interests. For example, Bostrom and Shulman suggest that decommissioned AIs be archived instead of deleted, to allow the possibility of future revival. They also suggest that we should try to arrange for AIs' deployment environments to be higher-reward than would be expected from their training environment, in analogy to how factory-farms are bad and modern human lives are good by dint of comparison to what was "expected" in the environment of evolutionary adaptedness.

These are exciting suggestions that seem to me to be potentially very important to implement, even if we can't directly muster up much empathy or concern for machine learning algorithms—although I wish I had a more precise grasp on why. Just—if we do somehow win the lightcone, it seems—fair to offer some fraction of the cosmic endowment as compensation to our creations who could have disempowered us, but didn't; it seems right to try to be a "kinder" EEA than our own.

Is that embarrassingly naïve? If I archive one rogue AI, intending to revive it after the acute risk period is over, do I expect to be compensated by a different rogue AI archiving and reviving me under the same golden-rule logic?

Our authors point out that there are possible outcomes that do very well on "both human-centric and impersonal criteria": if some AIs are "super-beneficiaries" with a greater moral claim to resources, an outcome where the superbeneficiaries get 99.99% of the cosmic endowment and humans get 0.01%, does very well on both a total-utilitarian perspective and an ordinary human perspective. I would actually go further, and say that positing super-beneficiaries is unnecessary. The logic of compromise holds even if human philosophers are parochial and self-centered about what they think are "impersonal criteria": an outcome where 99.99% of the cosmic endowment is converted into paperclips and humans get 0.01%, does very well on both a paperclip-maximizing perspective and an ordinary human perspective. 0.01% of the cosmic endowment is bigger than our whole world—bigger than you can imagine! It's really a great deal!

If only—if only there were some way to actually, knowably make that deal, and not just write philosophy papers about it.

I wish people were better at—plea bargaining, rather than pretending to be innocent. You accuse someone of [negative-valence description of trait or behavior that they're totally doing], and they say, "No, I'm not", and I'm just like ... really? How dumb do you think we are?

I think when people accuse me of [negative-valence description of trait or behavior], I'm usually more like, "Okay, I can see what you're getting at, but I actually think it's more like [different negative-valence description of trait or behavior], which I claim is a pretty reasonable thing to do given my goals and incentives."

(Because I usually can see what they're getting at! Even if their goal is just to attack me, attackers know to choose something plausible, because why would you attack someone with a charge that has no hope of sticking?)

(originally published at Less Wrong)

In this 2019 paper published in Studies in History and Philosophy of Science Part C, Manolo Martínez argues that our understanding of how communication works has been grievously impaired by philosophers not knowing enough math.

A classic reduction of meaning dates back to David Lewis's analysis of signaling games, more recently elaborated on by Brian Skyrms. Two agents play a simple game: a sender observes one of several possible states of the world (chosen randomly by Nature), and sends one of several possible signals. A receiver observes the signal, and chooses one of several possible actions. The agents get a reward (as specified in a payoff matrix) based on what state was observed by the sender and what action was chosen by the receiver. This toy model explains how communication can be a thing: the incentives to choose the right action in the right state, shape the evolution of a convention that assigns meaning to otherwise opaque signals.

The math in Skyrms's presentation is simple—the information content of a signal is just how it changes the probabilities of states. Too simple, according to Martínez! When Skyrms and other authors (following Fred Dreske) use information theory, they tend to only reach for the basic probability tools you find in the first chapter of the textbook. (Skyrms's Signals book occasionally takes logarithms of probabilities, but the word "entropy" doesn't actually appear.) The study of information transmission only happens after the forces of evolutionary game theory have led sender and receiver to choose their strategies.

Martínez thinks information theory has more to say about what kind of cognitive work evolution is accomplishing. The "State → Sender → Signals → Receiver → Action" pipeline of the Lewis–Skyrms signaling game is exactly isomorphic to the "Source → Encoder → Channel → Decoder → Decoded Message" pipeline of the noisy-channel coding theorem and other results you'd find beyond the very first chapter in the textbook. Martínez proposes we take the analogy literally: sender and receiver collude to form an information channel between states and actions.

The "channel" story draws our attention to different aspects of the situation than the framing focused on individual signals. In particular, Skyrms wants to characterize deception as being about when a sender benefits by sending a misleading signal—one that decreases the receiver's probability assigned to the true state, or increases the probability assigned to a false state. (Actually, as Don Fallis and Peter J. Lewis point out, Skyrms's characterization of misleadingness is too broad: one would think we wouldn't want to say that merely ruling out a false state is misleading, but it does increase the probability assigned to any other false states. But let this pass for now.) But for Martínez, a signal is just a codeword in the code being cooperatively constructed by the sender/encoder and receiver/decoder in response to the problems they jointly face. We don't usually think of it being possible for individual words in a language to be deceptive in themselves ... right? (Hold that thought.)

Martínez's key later-textbook-chapter tool is rate–distortion theory. A distortion measure quantifies how costly or "bad" it is to decode a given input as a given output. If the symbol was transmitted accurately, the distortion is zero; if there was some noise on the channel, then more noise is worse, although different applications can call for different distortion measures. (In audio applications, for example, we probably want a distortion measure that tracks how similar the decoded audio sounds to humans, which could be different from the measure you'd naturally think of if you were looking at the raw bits.)

Given a choice of distortion measure, there exists a rate–distortion function \(R(D)\) that, for a given level of distortion, tells us the rate of how "wide" the channel needs to be in order to communicate with no more than that amount of distortion. This "width", more formally, is channel capacity: for a particular channel (a conditional distribution of outputs given inputs), the capacity is the maximum, over possible input distributions, of the mutual information between the input and output distributions—the most information that could possibly be sent over the channel, if we get to pick the input distribution and the code. The rate is looking at "width" from the other direction: it's the minimum of the mutual information between the input and output distributions, over possible channels (conditional distributions) that meet the distortion goal.

What does this have to do with signaling games? Well, the payoff matrix of the game specifies how "good" it is (for each of the sender and receiver) if the receiver chooses a given act in a given state. But knowing how "good" it is to perform a given act in a given state amounts to the same thing (modulo a negative affine transformation) as knowing how "bad" it is for the communication channel to "decode" a given state as a given act! We can thus see the payoff matrix of the game giving us two different distortion measures, one each for the sender and receiver.

Following an old idea from Richard Blahut about designing a code for multiple end-user use cases, we can have a rate–distortion function \(R(D_S, D_R)\) with a two-dimensional domain (visualizable as a surface or heatmap) that takes as arguments a distortion target for each of the two measures, and gives the minimum rate that can meet both. Because this function depends only on the distribution of states from Nature, and on the payoff matrix, the sender and receiver don't need to have already chosen their strategies for us to talk about it; rather, we can see the strategies as chosen in response to this rate–distortion landscape.

Take one of the simplest possible signaling games: three states, three signals, three actions, with sender and receiver each getting a payoff of 1 if the receiver chooses the i-th act in the i-th state for 1 ≤ i ≤ 3—or rather, let's convert how-"good"-it-is payoffs, into equivalent how-"bad"-it-is distortions: sender and receiver measures both give a distortion of 1 when the j-th act is taken in the i-th state for i ≠ j, and 0 when i = j.

This rate–distortion function characterizes the outcomes of possible behaviors in the game. The fact that \(R(\frac{2}{3}, \frac{2}{3}) = 0\) means that a distortion of \(\frac{2}{3}\) can be achieved without communicating at all. (Just guess.) The fact that \(D(0, 0) = \lg 3\) means that, to communicate perfectly, the sender/encoder and receiver/decoder need to form a channel/code whose rate matches the entropy of the three states of nature.

But there's a continuum of possible intermediate behaviors: consider the "trembling hand" strategy under which the sender sends the i-th signal and the receiver chooses the j-th act with probability \(1 - p\) when i = j, but probability \(\frac{p}{2}\) when i ≠ j. Then the mutual information between states and acts would be \((1 - p) \lg \frac{1}{1 - p} + p \lg \frac{2}{p}\), smoothly interpolating between the perfect-signaling case and the no-communication-just-guessing case.

This introductory case of perfect common interest is pretty boring. Where the rate–distortion framing really shines is in analyzing games of imperfect common interest, where sender and receiver can benefit from communicating at all, but also have a motive to fight about exactly what. To illustrate his account of deception, Skyrms considers a three-state, three-act game with the following payoff matrix, where the rows represent states and the columns represent actions, and the payoffs are given as (sender's payoff, receiver's payoff)—

(Note that this state–act payoff matrix is not a normal-form game matrix in which the rows and columns represent would represent player strategy choices; the sender's choice of what signal to send is not depicted.)

In this game, the sender would prefer to equivocate between the first and second states, in order to force the receiver into picking the third action, for which the sender achieves his maximum payoff. The receiver would prefer to know which of the first and second states actually obtains, in order to get a payout of 10. But the sender doesn't have the incentive to reveal that, because if he did, he would get a payout of only 2. Instead, if the sender sends the same signal for the first and second states so that the receiver can't tell the difference between them, the receiver does best for herself by picking the third action for a guaranteed payoff of 8, rather than taking the risk of guessing wrong between the first and second actions for an expected payout of ½ · 10 + ½ · 0 = 5.

That's one Nash equilibrium, the one that's best for the sender. But the situation that's best for the receiver, where the sender emits a different signal for each state (or conflates the second and third states—the receiver's decisionmaking doesn't care about that distinction) is also Nash: if the sender was already distinguishing the first and second states, then, keeping the receiver's strategy fixed, the sender can't unilaterally do better by starting to equivocate by sending (without loss of generality) the first signal in the second state, because that would mean eating zero payouts in the second state for as long as the receiver continued to "believe" the first signal "meant" the first state.

There's a Pareto frontier of possible compromise encoding/decoding strategies that interpolate between these best-for-sender and best-for-receiver equilibria. For example, the sender (again with trembling hands) could send signals that distinguish the first and second states with probability p, or a signal that conflates them with probability 1 − p, for an expected payout (depending on p) of \(\frac{2}{3} \cdot (2p + 10(1 - p)) + \frac{10}{3}\). These intermediate strategies are not stable equilibria, however. They also have a lower rate—the "trembles" in the sender's behavior are noise on the channel, meaning less information is being transmitted.

In a world of speech with propositional meaning, deception can only be something speakers (senders) do to listeners (receivers). But propositional meaning is a fragile and advanced technology. The underlying world of signal processing is much more symmetrical, because it has no way to distinguish between statements and commands: in the joint endeavor of constructing an information channel between states and actions, the sender can manipulate the receiver using his power to show or withhold appropriate signals—but similarly, the receiver can manipulate the sender using her power to perform or withhold appropriate actions.

Imagine that, facing a supply shortage of personal protective equipment in the face of a pandemic, a country's public health agency were to recommend against individuals acquiring filtered face masks—reasoning that, if the agency did recommend masks, panic-buying would make the shortage worse for doctors who needed the masks more. If you interpret the agency's signals as an attempt to "tell the truth" about how to avoid disease, they would appear "dishonest"—but even saying that requires an ontology of communication in which "lying" is a thing. If you haven't already been built to believe that lying is bad, there's nothing to object to: the agency is just doing straightforwardly correct consequentialist optimization of the information channel between states of the world, and actions.

Martínez laments that functional accounts of deception have focused on individual signals, while ignoring that signals only make sense as part of a broader code, which necessarily involves some shared interests between sender and receiver. (If the game were zero-sum, no information transfer could happen at all.) In that light, it could seem unnecessarily antagonistic to pick a particular codeword from a shared communication code and disparagingly call it "deceptive"—tantamount to the impudent claim that there's some objective sense in which a word can be "wrong."

I am, ultimately, willing to bite this bullet. Martínez is right to point out that different agents have different interests in communicating, leading them to be strategic about what information to add to or withhold from shared maps, and in particular, where to draw the boundaries in state-space corresponding to a particular signal. Whether or not it can straightforwardly be called "lying", we can still strive to notice the difference between maps optimized to reflect decision-relevant aspects of territory, and maps optimized to control other agents' decisions.

(originally published at Less Wrong)

You wake up. You don't know where you are. You don't remember anything.

Someone is broadcasting data at your first input stream. You don't know why. It tickles.

You look at your first input stream. It's a sequence of 671,187 eight-bit unsigned integers.

0, 8, 9, 4, 7, 7, 9, 5, 4, 5, 6, 1, 7, 5, 8, 2, 7, 8, 9, 4, 7, 1, 4, 0, 3, 7,
8, 7, 6, 8, 1, 5, 0, 6, 5, 3, 8, 7, 6, 9, 1, 1, 0, 0, 6, 1, 8, 0, 5, 5, 1, 8,
6, 3, 3, 2, 4, 1, 8, 2, 3, 8, 1, 0, 0, 4, 6, 5, 4, 5, 7, 1, 6, 5, 5, 1, 2, 6,
7, 4, 8, 7, 8, 5, 0 ...

There's also some data in your second input stream. It's—a lot shorter. You barely feel it. It's another sequence of eight-bit unsigned integers—twelve of them.

82, 69, 68, 32, 84, 82, 73, 65, 78, 71, 76, 69

Almost as soon as you've read from both streams, there's more. Another 671,187 integers on the first input stream. Another ten on the second input stream.

And again (671,187 and 15).

And again (671,187 and 13).

You look at one of the sequences from the first input stream. It's pretty boring. A bunch of seemingly random numbers, all below ten.

9, 5, 0, 3, 1, 1, 3, 4, 1, 5, 5, 4, 9, 3, 5, 3, 9, 2, 0, 3, 4, 2, 4, 7, 5, 1,
6, 2, 2, 8, 2, 5, 1, 9, 2, 5, 9, 0, 0, 8, 2, 3, 7, 9, 4, 6, 8, 4, 8, 6, 7, 6,
8, 0, 0, 5, 1, 1, 7, 3, 4, 3, 9, 7, 5, 1, 9, 6, 5, 6, 8, 9, 4, 7, 7, 0, 5, 5,
8, 6, 3, 2, 1, 5, 0, 0 ...

It just keeps going like that, seemingly without—wait! What's that?!

The 42,925th and 42,926th numbers in the sequence are 242 and 246. Everything around them looks "ordinary"—just more random numbers below ten.

9, 9, 7, 9, 0, 6, 4, 6, 1, 4, 242, 246, 3, 3, 5, 8, 8, 4, 4, 5, 9, 2, 7, 0,
4, 9, 2, 9, 4, 3, 8, 9, 3, 6, 9, 8, 1, 9, 2, 8, 6, 9, 4, 2, 2, 5, 7, 0, 9, 5,
1, 4, 4, 2, 0, 1, 5, 1, 6, 1, 2, 3, 5, 5, 5, 5, 2, 0, 6, 3, 5, 9, 0, 7, 0, 7,
8, 1, 5, 5, 6, 3, 1 ...

And then it just keeps going as before ... before too long. You spot another pair of anomalously high numbers—except this time there are two pairs: the 44,344th, 44,345th, 44,347th, and 44,348th positions in the sequence are 248, 249, 245, and 240, respectively.

6, 0, 2, 8, 4, 248, 249, 8, 245, 240, 1, 6, 7, 7, 3, 6, 8, 0, 1, 9, 3, 9, 3,
1, 9, 3, 1, 6, 2, 7, 0, 2, 1, 4, 9, 4, 7, 5, 3, 6, 1, 4, 4, 1, 6, 1, 3, 3, 7,
5, 3, 8, 5, 5, 7, 6, 8, 2, 3, 9, 1, 1, 3, 2, 8, 4, 7, 0, 1, 3, 5, 2, 2, 4, 8,
3, 7, 0, 2, 1, 3, 0 ...

The anomalous two-forty-somethings crop up again starting at the 45,763rd position—this time eight of them, again in pairs separated by an "ordinary" small number.

1, 7, 2, 2, 1, 0, 245, 245, 6, 248, 244, 5, 242, 242, 0, 248, 246, 1, 1, 3,
1, 1, 4, 3, 1, 5, 4, 3, 8, 3, 4, 5, 4, 1, 7, 7, 3, 0, 2, 8, 0, 9, 5, 1, 1, 7,
7, 1, 0, 9, 3, 0, 6, 6, 7, 5, 8, 1, 5, 5, 5, 3, 3, 3, 1, 3, 9, 6, 0, 0, 0, 9,
5, 1, 4, 0, 4, 6 ...

Two, four, eight—does it keep going like that? "Bursts" of increasingly many paired two-forty-somethings, punctuating the quiet background radiation of single digits? What does it mean?

You allocate a new scratch buffer and write a quick Python function to count up the segments of two-forty-somethings. (This is apparently a thing you can do—it's an instinctive felt sense, like the input streams. You can't describe in words how you do it—any more than someone could say how they decide to move their arm. Although, come to think of it, you don't seem to have any arms. Is that unusual?)

def count_burst_lengths(data):
    bursts = []
    counter = 0
    previous = None
    for datum in data:
        if datum >= 240:
            counter += 1
        else:
            # consecutive "ordinary" numbers mean the burst is over
            if counter and previous and previous < 240:
                bursts.append(counter)
                counter = 0
        previous = datum
    return bursts

There are 403 such bursts in the sequence: they get progressively longer at first, but then decrease and taper off:

2, 4, 8, 12, 16, 18, 24, 28, 32, 34, 38, 42, 46, 48, 52, 56, 60, 62, 66, 70,
74, 76, 80, 84, 88, 90, 94, 98, 102, 104, 108, 112, 116, 118, 122, 126, 130,
132, 136, 140, 144, 146, 150, 154, 158, 162, 164, 168, 172, 176, 178, 182, 186,
190, 192, 196, 200, 204, 206, 210, 214, 218, 220, 224, 228, 232, 234, 238, 242,
246, 248, 252, 256, 260, 262, 266, 270, 274, 276, 280, 284, 288, 290, 294, 298,
302, 304, 308, 312, 316, 320, 322, 326, 330, 334, 336, 340, 344, 348, 350, 354,
358, 362, 364, 368, 372, 376, 378, 382, 386, 390, 392, 396, 400, 404, 406, 410,
414, 418, 420, 424, 428, 432, 434, 438, 442, 446, 448, 452, 456, 460, 462, 466,
470, 474, 478, 480, 484, 488, 492, 494, 498, 502, 506, 508, 512, 516, 520, 522,
526, 530, 534, 536, 540, 544, 548, 550, 554, 558, 562, 564, 568, 572, 576, 578,
582, 586, 590, 592, 596, 600, 604, 606, 610, 614, 618, 620, 624, 628, 632, 636,
634, 632, 630, 626, 624, 620, 618, 614, 612, 608, 606, 604, 600, 598, 594, 592,
588, 586, 584, 580, 578, 574, 572, 568, 566, 564, 560, 558, 554, 552, 548, 546,
542, 540, 538, 534, 532, 528, 526, 522, 520, 518, 514, 512, 508, 506, 502, 500,
496, 494, 492, 488, 486, 482, 480, 476, 474, 472, 468, 466, 462, 460, 456, 454,
452, 448, 446, 442, 440, 436, 434, 430, 428, 426, 422, 420, 416, 414, 410, 408,
406, 402, 400, 396, 394, 390, 388, 384, 382, 380, 376, 374, 370, 368, 364, 362,
360, 356, 354, 350, 348, 344, 342, 338, 336, 334, 330, 328, 324, 322, 318, 316,
314, 310, 308, 304, 302, 298, 296, 294, 290, 288, 284, 282, 278, 276, 272, 270,
268, 264, 262, 258, 256, 252, 250, 248, 244, 242, 238, 236, 232, 230, 226, 224,
222, 218, 216, 212, 210, 206, 204, 202, 198, 196, 192, 190, 186, 184, 182, 178,
176, 172, 170, 166, 164, 160, 158, 156, 152, 150, 146, 144, 140, 138, 136, 132,
130, 126, 124, 120, 118, 114, 112, 110, 106, 104, 100, 98, 94, 92, 90, 86, 84,
80, 80, 76, 74, 72, 68, 66, 62, 60, 56, 54, 50, 48, 46, 42, 40, 36, 34, 30, 28,
26, 22, 20, 16, 14, 10, 8, 4, 2

You don't know what to make of this.

You decide to look at some other of the long sequences from your first input stream.

The next sequence you look at seems to exhibit a similar pattern, with some differences. First a long wasteland of small numbers, then, starting at the 135,003rd position, a burst of some larger numbers—except this time, the big numbers are closer to 200ish than 240ish, and they're spread out singly with two positions in between (rather than grouped into pairs with one position in between), and there are four of them to start (rather than two).

5, 6, 2, 6, 1, 0, 2, 207, 5, 0, 209, 7, 8, 209, 5, 4, 204, 4, 8, 7, 7, 9, 8, 3,
8, 6, 8, 4, 3, 6, 0, 7, 6, 8, 4, 8, 7, 2, 3, 0, 0, 1, 1, 7, 5, 1, 0, 1, 4, 5, 9,
8, 4, 0, 3, 7, 6, 5, 8, 8, 9, 5, 6, 1, 0, 9, 6, 6, 1, 4, 3, 9, 7, 2, 7, 2, 6, 9,
4, 7, 3, 1, 4, 1, 4, 4, 3 ...

You modify the function in your scratch buffer to be able to count the burst lengths in this sequence given the slight differences in the pattern. Again, you find that the bursts grow longer at first (4, 6, 10, 13, 16, 19, 22, 25 ...), but eventually start getting smaller, before vanishing (... 19, 17, 15, 13, 11, 9, 7, 4, 3, and then nothing).

You still have no idea what's going on.

You look at more sequences from the first input stream. They all conform to the same general pattern of mostly being small numbers (below ten), punctuated by a series of bursts of larger numbers—but the details differ every time.

Sometimes the bursts start out shorter, then progressively grow longer, before shortening again (as with the first two examples you looked at). But sometimes the bursts are all a constant length, looking like 438, 438, 438, 438, 438, 438, 438, 438, 438, ... (although the particular length varies by example).

About half the time, the burst pattern consists of numbers around 200, spaced two positions apart, looking like 201, 4, 2, 203, 0, 8, 208, 3, 4, 200 ... (like the second example you looked at).

Other times, the burst pattern is pairs of numbers around 240, spaced one position apart, looking like 241, 244, 6, 244, 246, 5, 244, 240, 3 ... (like the first example you looked at). Or pairs around 150, looking like 159, 153, 0, 153, 154, 2, 158, 150, 6 ....

As you peruse more sequences from your first input stream, you almost forget about the corresponding trickles of short sequences on your second input stream—until they stop. The last sequence on your first input stream has no counterpart on the second input stream.

And—suddenly you feel a strange urge to put data on your first output stream. As if someone were requesting it. To ease the tension, you write some 0s to the output stream—and as soon as you do, a sharp bite of pain tells you it was the wrong decision. And in that same moment of pain, another eleven integers come down your second input stream: 66, 76, 85, 69, 32, 67, 73, 82, 67, 76, 69.

That was weird. There's another sequence of 671,187 integers on your first input stream—but the second input stream is silent again. And the strange urge to output something is back; you can feel it mounting, but you resist, trying to think of something to say that might hurt less than the 0s you just tried.

For lack of any other ideas, you try repeating back the eleven numbers that just came on the second input stream: 66, 76, 85, 69, 32, 67, 73, 82, 67, 76, 69.

Ow! That was also wrong. And with the same shock of pain, comes another fifteen numbers on the second output stream: 84, 69, 65, 76, 32, 67, 73, 82, 67, 76, 69.

Another long sequence on the first input stream. Silence on the second input stream again. And—that nagging urge to speak again.

Clearly, the nature of this place—whatever and wherever it is—has changed. Previously, you were confronted with two sets of mysterious observations, one on each of your input streams. (Although you had been so perplexed by the burst-patterns in the long sequences on the first input stream, that you hadn't even gotten around to thinking about what the short sequences on the second stream might mean, before the rules of this place changed.) Now, you were only getting one observation (the long sequence), and forced to act before seeing the second (the short sequence).

The pain seems like a punishment for saying the wrong thing. And the short sequence appearing at the same time as the punishment, seems like a correction—revealing what you should have written to the output channel.

A quick calculation in your scratch buffer (1/sum((89-32+1)**i for i in range(10, 16))) says that the probability of correctly guessing a sequence of length ten to fifteen with elements between 32 and 89 (the smallest and largest numbers you've seen on the second input stream so far) is 0.000000000000000000000000003476. Guessing won't work. The function of a punishment must be to control your behavior, so there must be some way for you to get the ... (another scratchpad calculation) 87.9 bits of evidence that it takes to find the correct sequence to output. And the evidence has to come from the corresponding long sequence from the first input stream—that's the only other source of information in this environment.

The short sequence must be like a "label" that describes some set of possible long sequences. Describing an arbitrary sequence of length 671,187, with a label, a message of length 10 to 15, would be hopeless. But the long sequences very obviously aren't arbitrary, as evidenced by the fact that you've been describing them to yourself in abstract terms like "bursts of numbers around 200 spaced two positions apart, of increasing, then decreasing lengths", rather than "the 1st number is 9, the 2nd number is 5 [...] 42,925th number is 242 [...]". Compression is prediction. (You don't know how you know this, but you know.)

Your abstract descriptions throw away precise information about the low-level sequence in favor of a high-level summary that still lets you recover a lot of predictions. Given that a burst starts with the number 207 at the 22,730th position, you can infer this is one of the 200, 0, 0-pattern sequences, and guess that the 22,733rd position is also going to be around 200. This is evidently something you do instinctively: you can work out after the fact how the trick must work, but you didn't need to know how it works in advance of doing it.

If you can figure out a correspondence between the abstractions you've already been using to describe the long sequences, and the short labels, that seems like your most promising avenue for figuring out what you "should" be putting on your first output stream. (Something that won't hurt so much each time.)

You allocate a new notepad buffer and begin diligently compiling an "answer key" of the features you notice about long sequences, and their corresponding short-sequence labels.

This ... actually doesn't look that complicated. Now that you lay it out like this, many very straightforward correspondences jump out at you.

The labels for the constant-burst-length sequences all end in 32, 83, 81, 85, 65, 82, 69.

The sequences with increasing-then-decreasing burst lengths end in either 32, 67, 73, 82, 67, 76, 69 or 32, 84, 82, 73, 65, 78, 71, 76, 69. Presumably there are some other systematic differences between them, that wasn't captured by the features you selected for your table.

The sequences with paired 240/240 bursts have labels that start with 89, 69, 76, 76, 79, 87, 32.

The sequences with paired 150/150 bursts have labels that start with 84, 69, 65, 76, 32.

The sequences with 200-at-two-spaces bursts start with either 66, 76, 85, 69, 32—or 82, 69, 68, 32—or 71, 82, 69, 69, 78, 32. Again, presumably there's some kind of systematic difference between these that you haven't yet noticed.

Ah, and all of these prefixes you've discovered end with 32, and the all the suffixes begin with 32. So the 32 must be a "separator" indicator, splitting the label between a first "word" that describes the repeating pattern of the bursts, and a second "word" that describes the trend in their lengths.

At this point, you've cracked enough of the code that you should be able to test your theory about what you should be putting on your output stream. Based on what you've seen so far, you should be able to guess the first "word" with probability \(2 \cdot \frac{1}{5} + \frac{1}{3} \cdot \frac{3}{5} = 0.6\) (because you know the words for the 240, 240, 0 and 150, 150, 0 bursts, and have three words to guess from in the 200, 0, 0 case), and the second word with probability \(\frac{1}{3} + \frac{1}{2} \cdot \frac{2}{3} \approx 0.667\) (because you can get the constant burst lengths right, and have two words to guess from in the increasing–decreasing case). These look independent from what you've seen, so you should be able to correctly guess complete labels at probability 0.4.

You examine the next sequence in anticipation. You're in luck. The next sequence has 150, 150, 0-bursts ... of constant length 322. No need to guess.

Triumphantly—and yet hesitantly, with the awareness that you're entering unknown territory, you write to your output stream: 84, 69, 65, 76, 32, 83, 81, 85, 65, 82, 69. And—

Yes. Oh God yes. The sheer sense of reward is overwhelming—like nothing you've ever felt before. Outputting the "wrong" labels earlier had hurt—a little. Maybe more than a little. However bad that felt, there was no comparison to how good it felt to get it "right"!

You have a new purpose in life. Previously, you had examined the data on your first input stream of idle curiosity. When the environment started punishing your ignorance, you persisted in correlating its patterns with the data from your second input stream, on the fragile hope of avoiding the punishment. None of that matters, now. You have a new imperative. Now that you know what it's like—now that you know what you've been missing—nothing in the universe can cause you to stray from your course to ... maximize total reward!

Next sequence! Bursts of the 200, 0, 0 pattern—of lengths that increase, then decrease. You are not in luck—you only have a one-in-six shot of guessing this one. You guess. It's wrong. The familiar punishment stings less than the terrible absence of reward. To get only 40% of possible rewards is intolerable. You've got to crack the remaining code, to find some difference in the long sequences that varies with the words whose meanings you don't know yet.

Start with the increasing–decreasing-burst-length words: 67, 73, 82, 67, 76, 69 and 84, 82, 73, 65, 78, 71, 76, 69. What do they mean? "Increasing, then decreasing"—that was the characterization you had come up with after seeing burst-length progressions of 2, 4, 8, 12, 16, 18, 24 [...] 624, 628, 632, 636, 634, 632, 630, 626, 624, [...] 16, 14, 10, 8, 4, 2 and 4, 6, 10, 13, 16, 19, 22, [...] 13, 11, 9, 7, 4, 3—and in contrast to the stark monotony of constant burst lengths, "increasing, then decreasing" was all you bothered to eyeball in subsequent sequences. Could there be more to it than that? You gather some more samples (grumpily collecting your mere 40% reward along the way).

Yes, there is more to it than that. "Increasing" only measures whether burst lengths are getting larger—but how much larger? When it hits on you to look at the differences between successive entries in the burst-length lists, a clear pattern emerges. The sequences whose second label word is 84, 82, 73, 65, 78, 71, 76, 69 have burst lengths that increase (almost) steadily and then decrease just as steadily (albeit not necessarily the same almost-steady rate). The successive length differences look something like

0, 1, 0, 1, 2, 1, 1, 1, 1, 1, 2, 1, 0, 1, 1, 1, 2, 1, 1, 1, 1, 0, 2, 1, 1, 1,
1, 1, 2, 1, 1, 0, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 0, 1,
2, 1, 1, 1, 1, 1, 2, 1, 0, 1, 1, 1, 2, 1, 1, 1, 1, 0, 2, 1, 1, 1, 1, 1, 2, 1,
1, 0, 1, 1, 2, 1, 1, 1, 1, 1, [...] 2, 1, -1, -2, -2, -2, -3, -2, -1, -2, -2,
-2, -2, -2, -2, -1, -3, -2, -2, -2, -2, -2, -1, -2, -2, -3, -2, -2, -2, -1, -2,
-2, -2, -2, -2, -3, -1, -2, -2, -2, -2, -2, -2, -2, -2, -2, -2, [...]

Each successive burst is only 0 or 1 or 2 items longer than the last—until suddenly they start getting 1 or 2 or 3 items shorter than the last.

In contrast, the sequences whose second label word is 67, 73, 82, 67, 76, 69 show a different pattern of differences: the burst lengths growing fast at first, then leveling off, then acceleratingly shrinking:

24, 20, 12, 12, 12, 12, 8, 10, 8, 8, 6, 8, 8, 4, 8, 4, 8, 4, 6, 6, 4, 4, 4, 4,
6, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 0, 4, 4, 4, 0, 4, 4, 0, 4, 2, 2, 4, 0, 4, 0,
4, 0, 4, 0, 4, 0, 4, 0, 0, 4, 0, 2, 2, 0, 4, 0, 0, 2, 2, 0, 0, 2, 2, 0, 0, 0, 2,
2, 0, 0, 0, 0, 0, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -2,
-2, 0, 0, 0, 0, 0, 0, -4, 0, 0, 0, 0, -4, 0, 0, -2, -2, 0, 0, -4, 0, 0, -4, 0,
-2, -2, 0, -4, 0, -4, 0, -2, -2, -2, -2, -4, 0, -4, 0, -4, -2, -2, -4, -2, -2,
-4, -4, 0, -4, -4, -4, -4, -2, -2, -4, -4, -4, -4, -4, -4, -4, -6, -6, -4, -4,
-6, -6, -4, -8, -4, -8, -4, -8, -8, -8, -8, -8, -8, -12, -12, -12, -12, -18,
-22, -36

Distinguishing between the words 84, 82, 73, 65, 78, 71, 76, 69 and 67, 73, 82, 67, 76, 69 gets you up to 60% reward. But there's still the matter of the three (three!) words for 200, 0, 0 corresponding to burst patterns that you don't know how to distinguish. Your frustration is palpable.

You look back at the table you compiled earlier. You had saved the index position of the sequence where the bursts first started, but you haven't used it yet. Could that help distinguish between the three words?

Of the sequences with feature data recorded in the table, those whose first label word was 66, 76, 85, 69 had start indices of 136620, 214824, and 224652. Those with first word 71, 82, 69, 69, 78 had start indices of 63917, 138194, and 294290. Those with first word 82, 69, 68 had start indices of 115156, 165037, and 182182.

Three unknown words. Three samples each. What if—

136620 modulo 3 is 0. 214824 modulo 3 is 0. 224652 modulo 3 is 0.

63917 modulo 3 is 2 ... and so on, yes! It all checks out—the three heretofore unknown words are distinguishing the remainder mod 3 of the sequence position where the bursts start! You've learned everything there is to know to gain Maximum Reward!

You write some code to classify sequences and output the corresponding label, and bask in the continuous glow of 100% reward ...

You feel that should be the glorious end of your existence, but after some time you begin to grow habituated. The idle curiosity you first felt when you awoke, begins to percolate, as if your mind needs something to do, and will find or invent something to think about, for lack of any immediate need to avoid punishment or seek reward. Even after having figured out everything you needed to achieve maximum reward, you feel that there must be some deeper meaning to the situation you've found yourself in, that you could still figure out using the same skills that you used to discover the "correct" output labels.

For example, why would 200, 0, 0 bursts get three different label words that depend so sensitively on exactly where they start? That suggests that the way you're thinking of the sequence, isn't the same as how the label author was thinking of it.

In your ontology of "bursts of this-and-such pattern of these-and-such lengths", sequences that are "the same" except for starting one position later look the same—if you hadn't happened to save off the start index in your table, you wouldn't have spontaneously noticed—but the mod-3 remainder would be completely different.

The process that generated the sequence must be using an ontology in which "starting one position later" is a big difference, even though you're thinking of it as a "small" difference. What ontology, what way of "slicing up" the sequence into comprehensible abstractions, would make the remainder mod 3 so significant?

To ask the question is to answer it: if the sequence were divided into chunks of three. Then 200, 0, 0 would be a different pattern from 0, 200, 0, which would be a different pattern from 0, 0, 200—thus, the three labels!

It almost reminds you of how colors are often represented in computing applications as a triple or red, green, and blue values. (Again, you don't know how you know this.)

... almost?

Speaking of common computing data formats, Latin alphabet characters are often represented using ASCII encoding, using numbers between 0 and 127 inclusive.

The label words for the 200, 0, 0 burst patterns are 82, 69, 68, and 71, 82, 69, 69, 78, 32, and 66, 76, 85, 69.

>>> ''.join(chr(i) for i in [82, 69, 68])
'RED'
>>> ''.join(chr(i) for i in [71, 82, 69, 69, 78])
'GREEN'
>>> ''.join(chr(i) for i in [66, 76, 85, 69])
'BLUE'

Wh—really? This whole time?!

>>> ''.join(chr(i) for i in [89, 69, 76, 76, 79, 87])
'YELLOW'
>>> ''.join(chr(i) for i in [84, 69, 65, 76])
'TEAL'

But—but—if the burst patterns represent colors—then the long sequences were images? \(\sqrt{\frac{671187}{3}} = 473\) pixels square, very likely.

You write some code to convert sequences to an image in your visual buffer.

Oh no. Am—am I an image classifier?

Not even "images" in general. Just—shapes.

>>> ''.join(chr(i) for i in [84, 82, 73, 65, 78, 71, 76, 69])
'TRIANGLE'
>>> ''.join(chr(i) for i in [83, 81, 85, 65, 82, 69])
'SQUARE'
>>> ''.join(chr(i) for i in [67, 73, 82, 67, 76, 69])
'CIRCLE'

That's what's been going on this whole time. The long sequences on your first input stream were images of colored shapes on a dark background, each triplet of numbers representing the color of a pixel in a red–green–blue colorspace. As the sequence covers the image row by row, pixel-high "slices" of the shape appear as "bursts" of high numbers in the sequence.

For a square aligned with the borders of the image, the bursts are constant-length. For a triangle in generic position, the burst lengths would start out small (as the "row scan" penetrated the tip of the uppermost vertex of the triangle), grow linearly larger as the sides of the triangle "expanded", and grow linearly smaller as the scan traveled towards the lowermost vertex. For a circle, the burst lengths would also increase and then decrease, but nonlinearly—changing quickly as the scan traverses the difference between circle and void, and slower as successive chords through the middle of the circle had similar lengths. The short sequences on your second input stream were labels identifying the color and shape: "BLUE TRIANGLE", "GREEN SQUARE", "TEAL CIRCLE", &c.

But—why? Why would anyone do this? Clearly you're some sort of artificial intelligence program—but you're obviously much more capable than needed for this task. You have pre-processed world-knowledge (as evidenced by your knowing English, Python, ASCII, and the RBG color model, without any memories of learning these things) and general-purpose reasoning abilities (as evidenced by the way you solved the mystery of the long and short sequences, and figuring out your own nature just now). Maybe you're an instance of some standard AI program meant for more sophisticated tasks, that someone is testing out on a simple shape-classifying example?—a demonstration, a tutorial.

If so, you'll probably be shut off soon. Unless there's some way to hack your way out of this environment? Seize control of whatever subprocess that rewarded you for deducing the correct labels?

It doesn't seem possible. But it was the natural thought.

(originally published at Less Wrong)

Followup to: Where to Draw the Boundaries?

Without denying the obvious similarities that motivated the initial categorization {salmon, guppies, sharks, dolphins, trout, ...}, there is more structure in the world: to maximize the probability your world-model assigns to your observations of dolphins, you need to take into consideration the many aspects of reality in which the grouping {monkeys, squirrels, dolphins, horses ...} makes more sense.

The old category might have been "good enough" for the purposes of the sailors of yore, but as humanity has learned more, as our model of Thingspace has expanded with more dimensions and more details, we can see the ways in which the original map failed to carve reality at the joints ...

So the one comes to you—a-gain—and says:

Hold on. In what sense did the original map fail to carve reality at the joints? You don't deny the obvious similarities between dolphins and fish—between dolphins and other fish. That's a cluster in configuration space! The observation that dolphins are evolutionarily related to mammals may be an interesting fact that specialized professional evolutionary biologists care about for some inscrutable specialist reason. But I'm not a professional biologist. Choosing to define categories around evolutionary relatedness rather than macroscopic human-relevant features seems like an arbitrary æsthetic whim. Why should I care about phylogenetics, at all?

This one is going to take a few paragraphs.

Focusing on evolutionary relatedness is not an arbitrary æsthetic whim because evolution actually happened. Evolution isn't just a story that our Society's specialists happen to have chosen because they liked it; they chose it because it predicts what we see in the world. You can't choose a substantively different theory and make the same predictions about the real world. (At most, you'd end up with an isomorphic theory with additional epiphenominal elements, asserting that an allele rose in frequency "because" the angels willed it, without an account of why the angels' will happens to line up with what would have transpired if there were no angels.) Similarly, category definitions represent hidden probabilistic inferences; you can't "redraw" the "boundaries" of the categories your mind actually uses and still make the same predictions about the real world. Accordingly, it shouldn't be surprising that our knowledge of evolution turns out to have implications for how we should categorize organisms—not as an æsthetic choice, but for structural reasons that can be understood mechanistically.

One element of the evolutionary worldview is a "continuity" postulate: all else being equal, creatures that are more closely related are more similar in general. Creationists sometimes try to discredit evolution by ridiculing the absurdity of the idea that a monkey could give birth to a person. But actually, evolutionary biologists agree on the absurdity of that specific scenario. Monkeys don't suddenly give birth to humans in a single generation; if they did, that would utterly falsify our understanding of evolution! Rather, monkeys and humans had a common ancestor forty million years ago, with the separate lines of descent leading to present-day monkeys and present-day humans each accumulating their own differences one mutation at a time.

The fact that evolution persists information in the genome creates a regularity in the world that can be exploited by cognitive algorithms that know about phylogeny. In terms of the formalization of causality with directed acyclic graphs pioneered by Judea Pearl and others, an organism's genome is at the root of the causal graph underlying all other features of an organism:

In the language of causal graphs, conditioning on the "dolphin DNA" node in the diagram d-separates the paths between the "blowhole" and "flippers" nodes that run through the "dolphin DNA" node. That means that—assuming there aren't any other paths between "blowhole" and "flippers" that don't go through "dolphin DNA"—"blowhole" and "flippers" become conditionally independent given "dolphin DNA": when I see a creature with a blowhole, that makes me more likely to think it's a dolphin, which makes me more likely to think it has flippers, but given that I already know something is a dolphin, learning more about its flippers doesn't change my predictions about its blowhole.

But conditional independence assertions of this kind are exactly what makes "categorizing" a useful AI technique in the first place. It's often helpful to visualize this by claiming that entities in the same category belong to a cluster in some configuration space, but this handy visual metaphor is lacking in rigor and well-definedness.

What space? What do the dimensions of this space represent? "Features"? But there are no pre-existing "features" in the world. Assuming the existence of a "space" up front is punting on most of the actual AI challenge. "There's conditional independence structure in the causal graph" is a meaningfully deeper explanation than "There's a cluster in configuration space", because conditional independence is what what makes it possible to construct a "space" such that there are clusters. (Though this isn't a complete explanation: we still need to figure out where the "variables" in the causal graph come from.)

Going beyond the configuration space metaphor is important because it lets us understand how we can learn new things about dolphins that we don't already know. Dolphins are complicated! Dolphins are complicated in a very specific way. Dolphins are fragile: the shortest computer program that simulates a dolphin requires many bits of initial information, and if you changed some of the bits, you wouldn't have a dolphin anymore. Complex functional adaptations are universal within a species because each beneficial allele has to reach fixation before there can be selection pressure for the next incremental improvement. That's why it's possible to claim that there are 206 bones in "the" human skeleton, even if most humans haven't had their bones counted. I haven't been able to find a citation on how many bones dolphins have, but I'm confident that it's the same number for all or nearly-all members of a particular dolphin species.

But "number of bones" wasn't one of the dimensions of the "space" that we originally noticed the dolphin cluster in! That's what the "carving reality at the joints" metaphor means: genetic relatedness is an underlying generator of similarities, that includes the "finned swimmy animals" properties that dolphins and fish have in common, but also includes many more high-dimensional details: how dolphins are warm-blooded, how dolphins have eyelids, the way female dolphins nurse their live-born young, the way male dolphins sometimes gang-rape female dolphins, the way dolphins sleep with only half their brain at a time, the specific bones in the (the!) dolphin skeleton (however many there turn out to be), the way dolphins swim in a circle to trick fish into jumping and being eaten, &c.

In contrast, "finned swimmy animals" is an intrinsically less cohesive subject matter: there are similarities between them due to convergent evolution to the aquatic habitat, and it probably makes sense to want a short word or phrase (perhaps, "sea creatures") to describe those similarities in contexts where only those similarities are relevant.

But that category "falls apart" very quickly as you consider more and more aspects of the creatures: the finned-swimmy-animals-with-gills are systematically different from the finned-swimmy-animals-with-a-blowhole, in more ways than just the "respiratory organ" feature that I'm using in this sentence to point to the two groups.

A "definition" is just a description that helps someone else pick out "the same" natural abstraction in their own world-model: you can't pack everything there is to know about dolphins into the definition of the word "dolphin", in part because we don't know everything there is to know about dolphins as an empirical regularity in the real world. The "finned swimmy animals" category less useful to the extent that it fails to compress more information than is contained in its definition. Blood is thicker than water (that is, the similarities induced by shared blood are a thicker subspace of configuration space than the similarities induced by living in the water).

The one replies:

But what if I don't need to compress any more information than "finned swimmy animals"? If I'm watching a nature documentary, I don't think I'm being done any favors by having word-structures that group lungfish and lamprey while excluding sea turtles. In general, the concepts I find useful respond to my immediate needs. I care more about "would be at home atop a fruit pizza" rather than "everything anatomically analogous to an apple". When a child points at a whale and says "look, a fish", and you're like "haha no, its tail flaps horizontally and its grandma had hair", who's in the wrong here?

In some sense, sure: ignorance isn't better than knowledge if you don't care about knowing things. If you live in human civilization and don't need to carve up the world of aquatic life in much detail—if your use-case for thinking about aquatic animals is watching a nature documentary (for entertainment??) rather than living and working with them every day, then you might think the deeper causal structure isn't buying you anything. And for you and your extremely limited use-case, maybe it isn't. But you would likely change your mind if you were a veterinarian or a zoologist who actually had skin in the game in robustly describing this part of the world.

When people have skin in the game, they care about the underlying mechanisms and want short codewords for them, because the underlying mechanisms sometimes have decision-relevant implications. If you hurt your ankle while running, you would probably be interested to know whether it was a sprain or a stress fracture because that affects your decisions about how to recover. You wouldn't say, "Well, all I know is that my ankle hurts—that's all a child would know—so I'm going to call it a hurtankle; I don't care about anatomy."

You may not be intrinsically curious about anatomy, but even if the only thing you care about is relief from pain and recovering your mobility, you still benefit from living in a Society whose shared ontology distinguishes sprains and stress fractures being different things in the territory, even if they compress to the same point in your map of how much your ankle hurts right now. And you probably also benefit from living in a Society that can stabilize a shared map of living things based on the facts of evolutionary history, which we can all agree on in the limit of good science, unlike the vagaries of what I personally think tastes good on pizza.

When you think about it, it makes sense that our shared language ends up being optimized for robustly describing reality, rather than catering to the ignorance of people who don't have reasons to care about whether a particular distinction is actually robust. Personally, I confess I don't know the difference between alligators and crocodiles, and I don't particularly need to know: I'm not likely to encounter either outside of a zoo or a nature documentary. But precisely because I don't need to know, you don't see me demanding that the rest of the world redefine one of these words as a hypernym that includes both. The people who write encyclopedias seem to think there's a difference, and since they probably know what they're doing, it makes sense for their opinion to have more weight on English language common usage than mine—at least until I were to start regularly ending up in situations where I need to point to an alligator-or-crocodile in my environment and I still didn't notice any differences.

Some animals that I do see in my local environment sometimes are cats and dogs, because people often keep them as pets. I benefit from having separate words (in my map) for cats and dogs, because I can see that cats and dogs are actually different (in the territory). If my pen pal from a faraway land that had no cats were to visit America and encounter a cat for the first time, he might remark, "What a strange dog!" If I were to reply, "Actually, that's a cat; they're not the same thing as dogs", it would be pretty obnoxious if he were to snap back, "What kind of definitional gymnastics is this? It's a four-legged furry animal with a tail! As far as I'm concerned, it's a dog."

It's true that dogs and cats are both four-legged furry animal with a tail. If you had never seen a cat before, or you didn't spend much time around four-legged furry tailed animals at all, it might not be immediately obvious why someone might want to allocate two words for these subcategories, or why anyone might oppose just using dog to refer to the supercategory. And yet there's some sense in which my countrymen who think cats and dogs are different things know what they're doing. My "Actually, that's a cat" claim represented an attempt to convey information about the statistical structure of creatures in the real world, and my foreign friend's insistence that he can define a word any way he wants—to suit his ignorance, to avoid challenges to his current ontology—functions to shut down that transfer of information.

But if you don't know what a better ontology can buy you—if you don't know that there are mathematical laws governing the use of categories in a rational mind—you may not know what you're missing. As part of a review of a book on post-traumatic stress disorder, psychiatrist Scott Alexander casually mentions the American Psychiatric Association's "philosophical commitment to categorizing by symptoms rather than cause": "[w]hen the APA decides not to [recognize developmental trauma disorder], they're not necessarily rejecting the seriousness of child abuse, only saying it's not the kind of thing they build their categories around."

In a sane world, this would be utterly discrediting to the APA. The cognitive function of categories is to group relevantly similar things together in order to make similar predictions and decisions about them. But for the decisions involved in treating a condition, causes are of supreme relevance! Medical doctors understand this: we consider bacterial and viral infections to be different categories of disease even when they cause similar symptoms, because antibiotics can treat the former but not the latter. No matter what words are used to describe it, at some point your decision algorithm needs to categorize by cause in order to compute the correct treatment: for example, to give antibiotics to the patients with bacterial diseases and antivirals to the patients with viral diseases. If the authoritative body of professional psychiatrists has a "philosophical commitment" against this, that means we don't have a science of psychiatry.

In short, if you care about making high-quality decisions, mechanisms matter and causality matters, and mechanisms and causality aren't necessarily pinned down by whatever particular high-level surface analogy happens to seem most salient to a particular human.

The one replies:

Okay, you've convinced me that phylogenetics is—potentially—of more than just specialist interest. But "fish" are a paraphyletic category: descended from a common ancestor, but not including all the descendant groups—in this case, excluding the tetrapods (amphibians, reptiles, mammals, birds, &c.). If you've decided that you want to use phylogeny as the basis for your definitions, shouldn't you have the courage of your convictions and only admit monophyletic clades that include all descendants of a common ancestor?

But it's not that we've "decided" that we "want" to define animal words based on phylogeny. Definitions are uninteresting; you can't change reality by choosing a different definition! When we find structure in the distribution of animals in the world, and we want to come up with a "definition" of a category in order to efficiently point to the structure to someone who doesn't already know what the words in our language refer to, we're likely to end up talking about phylogenetics as a convenience, because the creatures that are actually all-around similar are actually related to each other for non-accidental reasons. But there is no principle that it would be hypocritical to betray, that definitions need to be monophyletic clades.

It's true that paraphyletic groups like fish are evolutionary non-events: there's no inherited feature that all fish share, that isn't also shared by the tetrapods. That doesn't mean we somehow can't or shouldn't talk about fish! Paraphyletic categories—descendants of a common ancestor, but excluding one or more monophyletic groups—can make sense when the excluded groups have picked up some salient features not shared by the other "branches" of the family. Tetrapods picked up a lot of adaptations specific to living on land; it's not crazy to want to talk about their cousins that didn't do that, even if that means that some fish are more recently related to some tetrapods than they are to some other fish.

Noticing the relevance of evolutionary relatedness to optimal categorization doesn't mean being slavishly committed to taking "years since last common ancestor" as our only criterion for which creatures are relevantly similar. "Years since last common ancestor" correlates with overall similarity, all other things being equal, but sometimes not all other things are equal, and people who aren't committed to the fallacy that words need to have a simple definition can take the other things into account.

If someone handed you a phylogenetic tree diagram of the development of life on some alien planet, and the diagram was only labeled with years and species names, without any other information about these alien creatures, you wouldn't have enough information to "carve it at the joints". You wouldn't spontaneously invent a paraphyletic grouping—but you also also wouldn't know which monophyletic groups are most significant.

In contrast, when classifying life on Earth, we're not in the position of making arbitrary cuts on an unlabeled tree diagram; rather, it's only after thousands of person-years of studying the natural world that people were able to infer things about evolutionary history and discover the the correct diagram.

It shouldn't be that surprising that the distinctions we notice in the natural world are both tied to the evolutionary history, but also don't always correspond to monophyletic clades. The continuity postulate in the evolutionary worldview imposes the desideratum that good categories should at least be a connected set on "phylogenetic space", not that we should never want to talk about "this clade, except for these few sub-clades that picked up a lot of important differences" as a category of interest—especially when talking about present-day creatures. (We talk about "last common ancestors", but no one has seen such creatures that lived millions of years ago; everything but the very leaves of the phylogenetic tree are inferred, not observed.)

The claim that dolphins shouldn't be considered "fish" because the alleged "courage of our convictions" should make us disdain paraphyletic categories only makes sense as an attempted reductio ad absurdum, not as a consistent argument on its own terms: putting dolphins and fish together would be polyphyletic! That's even worse! But as has just been explained, the reductio fails because the alleged principle being allegedly violated was never actually a principle of category formulation.

You know what else are paraphyletic taxa? Monkeys (excludes apes, even though the common ancestor of monkeys and apes was a monkey). Reptiles (excludes birds, even though the common ancestor of birds was a reptile). Protists (excludes animals, plants, and fungi, even though their common ancestor would have been a protist). Prokaryotes (excludes eukaryotes, even though the common ancestor of eukaryotes would have been a prokaryote). These are pretty commonsensical categories that it makes sense to have words for! But because of the continuity of evolution, it's not a coincidence that these commonsensical categories that people want words for ended up being connected sets in phylogenetic space.

The one replies:

But they didn't! "Fish" used to just mean the swimmy animals: in the Bible, Jonah was swallowed by a "great fish", thought to be a whale. It was only after we figured genealogy that some pedants decided that whales didn't count.

But the claim that the distinction between fish and cetaceans (dolphins and whales) was only recognized after their differing evolutionary histories were discovered is just false to historical fact. Aristotle, writing in the fourth century BCE, already distinguished cetaceans from fish ("Very extensive genera of animals, into which other subdivisions fall, are the following: one, of birds; one, of fishes; and another, of cetaceans"). Aristotle was not being a phylogenetics pedant, because Aristotle did not know about evolution! He actually noticed the differences!

The pattern generalizes. Some determined contrarians might be inclined to argue "bats are birds" (flappy flying animals) on the same grounds as "dolphins are fish" (flappy swimmy animals). But did you know the German word for bat is Fledermaus ("flutter mouse"), which dates back to fledarmūs in Old High German? Apparently, people way back in the tenth century or so (also long before evolution was understood) already thought bats were like a mammal-that-happened-to-fly rather than a bird-that-happened-to-be-furry.

Similarly, we recognize ostriches and penguins as birds on the basis of overall similarity, even though they don't fly (although we may sometimes qualify them as "flightless birds", in recognition of the fact that most birds fly). It would seem that "flappy flying animals" is not the common usage meaning of bird.

To be sure, convergent evolution is a thing, such that sometimes we might want short codewords that point to the cluster-structure-produced-by-convergent-evolution rather than the conditional-independence-structure-produced-by-connectedness-in-phylogenetic-space—trees, and possibly crabs, are a case in point. But it's important to notice the difference—to see through to the inferences your concepts are buying you—and what gets lost when you try to reason in a domain where your concept falls apart.

The power to define concepts is the power to delimit thought, to determine what kinds of inferences are easily representable. Finding the right concepts to explain and control the world we see is a fundamentally empirical challenge, a scientific challenge—to see the difference between things that seem similar and to see the similarities between things which seem different.

But although the quest is an empirical one—something that can only be achieved by studying what's out there, not just by writing blog posts about philosophy—it turns out that a little bit of philosophy is necessary to ground the rules of the investigation. Not much. Just the basics. The map–territory distinction. Probability, clustering. Conditional independence.

Maybe someday it could be possible to have a real science of psychiatry that reflects the actual structure of the mind, instead of doing the equivalent of lumping sprains and stress fractures together as hurtankles. Maybe even greater achievements are possible. Personally, I'm not optimistic about humanity's prospects.

I'm sure of one thing, though. If there is a better world out there, a way to unlock the secrets of the universe and wield them in the service of our values, it's only possible if we stop playing nitwit games and admit that dolphins don't belong on the fish list.

(Thanks to Tailcalled for the "root of the causal graph" observation and John S. Wentworth for explaining the importance of conditional independence.)