An Algorithmic Lucidity

(originally published at Less Wrong)

In this 2019 paper published in Studies in History and Philosophy of Science Part C, Manolo Martínez argues that our understanding of how communication works has been grievously impaired by philosophers not knowing enough math.

A classic reduction of meaning dates back to David Lewis's analysis of signaling games, more recently elaborated on by Brian Skyrms. Two agents play a simple game: a sender observes one of several possible states of the world (chosen randomly by Nature), and sends one of several possible signals. A receiver observes the signal, and chooses one of several possible actions. The agents get a reward (as specified in a payoff matrix) based on what state was observed by the sender and what action was chosen by the receiver. This toy model explains how communication can be a thing: the incentives to choose the right action in the right state, shape the evolution of a convention that assigns meaning to otherwise opaque signals.

The math in Skyrms's presentation is simple—the information content of a signal is just how it changes the probabilities of states. Too simple, according to Martínez! When Skyrms and other authors (following Fred Dreske) use information theory, they tend to only reach for the basic probability tools you find in the first chapter of the textbook. (Skyrms's Signals book occasionally takes logarithms of probabilities, but the word "entropy" doesn't actually appear.) The study of information transmission only happens after the forces of evolutionary game theory have led sender and receiver to choose their strategies.

Martínez thinks information theory has more to say about what kind of cognitive work evolution is accomplishing. The "State → Sender → Signals → Receiver → Action" pipeline of the Lewis–Skyrms signaling game is exactly isomorphic to the "Source → Encoder → Channel → Decoder → Decoded Message" pipeline of the noisy-channel coding theorem and other results you'd find beyond the very first chapter in the textbook. Martínez proposes we take the analogy literally: sender and receiver collude to form an information channel between states and actions.

The "channel" story draws our attention to different aspects of the situation than the framing focused on individual signals. In particular, Skyrms wants to characterize deception as being about when a sender benefits by sending a misleading signal—one that decreases the receiver's probability assigned to the true state, or increases the probability assigned to a false state. (Actually, as Don Fallis and Peter J. Lewis point out, Skyrms's characterization of misleadingness is too broad: one would think we wouldn't want to say that merely ruling out a false state is misleading, but it does increase the probability assigned to any other false states. But let this pass for now.) But for Martínez, a signal is just a codeword in the code being cooperatively constructed by the sender/encoder and receiver/decoder in response to the problems they jointly face. We don't usually think of it being possible for individual words in a language to be deceptive in themselves ... right? (Hold that thought.)

Martínez's key later-textbook-chapter tool is rate–distortion theory. A distortion measure quantifies how costly or "bad" it is to decode a given input as a given output. If the symbol was transmitted accurately, the distortion is zero; if there was some noise on the channel, then more noise is worse, although different applications can call for different distortion measures. (In audio applications, for example, we probably want a distortion measure that tracks how similar the decoded audio sounds to humans, which could be different from the measure you'd naturally think of if you were looking at the raw bits.)

Given a choice of distortion measure, there exists a rate–distortion function \(R(D)\) that, for a given level of distortion, tells us the rate of how "wide" the channel needs to be in order to communicate with no more than that amount of distortion. This "width", more formally, is channel capacity: for a particular channel (a conditional distribution of outputs given inputs), the capacity is the maximum, over possible input distributions, of the mutual information between the input and output distributions—the most information that could possibly be sent over the channel, if we get to pick the input distribution and the code. The rate is looking at "width" from the other direction: it's the minimum of the mutual information between the input and output distributions, over possible channels (conditional distributions) that meet the distortion goal.

What does this have to do with signaling games? Well, the payoff matrix of the game specifies how "good" it is (for each of the sender and receiver) if the receiver chooses a given act in a given state. But knowing how "good" it is to perform a given act in a given state amounts to the same thing (modulo a negative affine transformation) as knowing how "bad" it is for the communication channel to "decode" a given state as a given act! We can thus see the payoff matrix of the game giving us two different distortion measures, one each for the sender and receiver.

Following an old idea from Richard Blahut about designing a code for multiple end-user use cases, we can have a rate–distortion function \(R(D_S, D_R)\) with a two-dimensional domain (visualizable as a surface or heatmap) that takes as arguments a distortion target for each of the two measures, and gives the minimum rate that can meet both. Because this function depends only on the distribution of states from Nature, and on the payoff matrix, the sender and receiver don't need to have already chosen their strategies for us to talk about it; rather, we can see the strategies as chosen in response to this rate–distortion landscape.

Take one of the simplest possible signaling games: three states, three signals, three actions, with sender and receiver each getting a payoff of 1 if the receiver chooses the i-th act in the i-th state for 1 ≤ i ≤ 3—or rather, let's convert how-"good"-it-is payoffs, into equivalent how-"bad"-it-is distortions: sender and receiver measures both give a distortion of 1 when the j-th act is taken in the i-th state for i ≠ j, and 0 when i = j.

This rate–distortion function characterizes the outcomes of possible behaviors in the game. The fact that \(R(\frac{2}{3}, \frac{2}{3}) = 0\) means that a distortion of \(\frac{2}{3}\) can be achieved without communicating at all. (Just guess.) The fact that \(D(0, 0) = \lg 3\) means that, to communicate perfectly, the sender/encoder and receiver/decoder need to form a channel/code whose rate matches the entropy of the three states of nature.

But there's a continuum of possible intermediate behaviors: consider the "trembling hand" strategy under which the sender sends the i-th signal and the receiver chooses the j-th act with probability \(1 - p\) when i = j, but probability \(\frac{p}{2}\) when i ≠ j. Then the mutual information between states and acts would be \((1 - p) \lg \frac{1}{1 - p} + p \lg \frac{2}{p}\), smoothly interpolating between the perfect-signaling case and the no-communication-just-guessing case.

This introductory case of perfect common interest is pretty boring. Where the rate–distortion framing really shines is in analyzing games of imperfect common interest, where sender and receiver can benefit from communicating at all, but also have a motive to fight about exactly what. To illustrate his account of deception, Skyrms considers a three-state, three-act game with the following payoff matrix, where the rows represent states and the columns represent actions, and the payoffs are given as (sender's payoff, receiver's payoff)—

(Note that this state–act payoff matrix is not a normal-form game matrix in which the rows and columns represent would represent player strategy choices; the sender's choice of what signal to send is not depicted.)

In this game, the sender would prefer to equivocate between the first and second states, in order to force the receiver into picking the third action, for which the sender achieves his maximum payoff. The receiver would prefer to know which of the first and second states actually obtains, in order to get a payout of 10. But the sender doesn't have the incentive to reveal that, because if he did, he would get a payout of only 2. Instead, if the sender sends the same signal for the first and second states so that the receiver can't tell the difference between them, the receiver does best for herself by picking the third action for a guaranteed payoff of 8, rather than taking the risk of guessing wrong between the first and second actions for an expected payout of ½ · 10 + ½ · 0 = 5.

That's one Nash equilibrium, the one that's best for the sender. But the situation that's best for the receiver, where the sender emits a different signal for each state (or conflates the second and third states—the receiver's decisionmaking doesn't care about that distinction) is also Nash: if the sender was already distinguishing the first and second states, then, keeping the receiver's strategy fixed, the sender can't unilaterally do better by starting to equivocate by sending (without loss of generality) the first signal in the second state, because that would mean eating zero payouts in the second state for as long as the receiver continued to "believe" the first signal "meant" the first state.

There's a Pareto frontier of possible compromise encoding/decoding strategies that interpolate between these best-for-sender and best-for-receiver equilibria. For example, the sender (again with trembling hands) could send signals that distinguish the first and second states with probability p, or a signal that conflates them with probability 1 − p, for an expected payout (depending on p) of \(\frac{2}{3} \cdot (2p + 10(1 - p)) + \frac{10}{3}\). These intermediate strategies are not stable equilibria, however. They also have a lower rate—the "trembles" in the sender's behavior are noise on the channel, meaning less information is being transmitted.

In a world of speech with propositional meaning, deception can only be something speakers (senders) do to listeners (receivers). But propositional meaning is a fragile and advanced technology. The underlying world of signal processing is much more symmetrical, because it has no way to distinguish between statements and commands: in the joint endeavor of constructing an information channel between states and actions, the sender can manipulate the receiver using his power to show or withhold appropriate signals—but similarly, the receiver can manipulate the sender using her power to perform or withhold appropriate actions.

Imagine that, facing a supply shortage of personal protective equipment in the face of a pandemic, a country's public health agency were to recommend against individuals acquiring filtered face masks—reasoning that, if the agency did recommend masks, panic-buying would make the shortage worse for doctors who needed the masks more. If you interpret the agency's signals as an attempt to "tell the truth" about how to avoid disease, they would appear "dishonest"—but even saying that requires an ontology of communication in which "lying" is a thing. If you haven't already been built to believe that lying is bad, there's nothing to object to: the agency is just doing straightforwardly correct consequentialist optimization of the information channel between states of the world, and actions.

Martínez laments that functional accounts of deception have focused on individual signals, while ignoring that signals only make sense as part of a broader code, which necessarily involves some shared interests between sender and receiver. (If the game were zero-sum, no information transfer could happen at all.) In that light, it could seem unnecessarily antagonistic to pick a particular codeword from a shared communication code and disparagingly call it "deceptive"—tantamount to the impudent claim that there's some objective sense in which a word can be "wrong."

I am, ultimately, willing to bite this bullet. Martínez is right to point out that different agents have different interests in communicating, leading them to be strategic about what information to add to or withhold from shared maps, and in particular, where to draw the boundaries in state-space corresponding to a particular signal. Whether or not it can straightforwardly be called "lying", we can still strive to notice the difference between maps optimized to reflect decision-relevant aspects of territory, and maps optimized to control other agents' decisions.

(originally published at Less Wrong)

Followup to: Where to Draw the Boundaries?

Without denying the obvious similarities that motivated the initial categorization {salmon, guppies, sharks, dolphins, trout, ...}, there is more structure in the world: to maximize the probability your world-model assigns to your observations of dolphins, you need to take into consideration the many aspects of reality in which the grouping {monkeys, squirrels, dolphins, horses ...} makes more sense.

The old category might have been "good enough" for the purposes of the sailors of yore, but as humanity has learned more, as our model of Thingspace has expanded with more dimensions and more details, we can see the ways in which the original map failed to carve reality at the joints ...

So the one comes to you—a-gain—and says:

Hold on. In what sense did the original map fail to carve reality at the joints? You don't deny the obvious similarities between dolphins and fish—between dolphins and other fish. That's a cluster in configuration space! The observation that dolphins are evolutionarily related to mammals may be an interesting fact that specialized professional evolutionary biologists care about for some inscrutable specialist reason. But I'm not a professional biologist. Choosing to define categories around evolutionary relatedness rather than macroscopic human-relevant features seems like an arbitrary æsthetic whim. Why should I care about phylogenetics, at all?

This one is going to take a few paragraphs.

Focusing on evolutionary relatedness is not an arbitrary æsthetic whim because evolution actually happened. Evolution isn't just a story that our Society's specialists happen to have chosen because they liked it; they chose it because it predicts what we see in the world. You can't choose a substantively different theory and make the same predictions about the real world. (At most, you'd end up with an isomorphic theory with additional epiphenominal elements, asserting that an allele rose in frequency "because" the angels willed it, without an account of why the angels' will happens to line up with what would have transpired if there were no angels.) Similarly, category definitions represent hidden probabilistic inferences; you can't "redraw" the "boundaries" of the categories your mind actually uses and still make the same predictions about the real world. Accordingly, it shouldn't be surprising that our knowledge of evolution turns out to have implications for how we should categorize organisms—not as an æsthetic choice, but for structural reasons that can be understood mechanistically.

One element of the evolutionary worldview is a "continuity" postulate: all else being equal, creatures that are more closely related are more similar in general. Creationists sometimes try to discredit evolution by ridiculing the absurdity of the idea that a monkey could give birth to a person. But actually, evolutionary biologists agree on the absurdity of that specific scenario. Monkeys don't suddenly give birth to humans in a single generation; if they did, that would utterly falsify our understanding of evolution! Rather, monkeys and humans had a common ancestor forty million years ago, with the separate lines of descent leading to present-day monkeys and present-day humans each accumulating their own differences one mutation at a time.

The fact that evolution persists information in the genome creates a regularity in the world that can be exploited by cognitive algorithms that know about phylogeny. In terms of the formalization of causality with directed acyclic graphs pioneered by Judea Pearl and others, an organism's genome is at the root of the causal graph underlying all other features of an organism:

In the language of causal graphs, conditioning on the "dolphin DNA" node in the diagram d-separates the paths between the "blowhole" and "flippers" nodes that run through the "dolphin DNA" node. That means that—assuming there aren't any other paths between "blowhole" and "flippers" that don't go through "dolphin DNA"—"blowhole" and "flippers" become conditionally independent given "dolphin DNA": when I see a creature with a blowhole, that makes me more likely to think it's a dolphin, which makes me more likely to think it has flippers, but given that I already know something is a dolphin, learning more about its flippers doesn't change my predictions about its blowhole.

But conditional independence assertions of this kind are exactly what makes "categorizing" a useful AI technique in the first place. It's often helpful to visualize this by claiming that entities in the same category belong to a cluster in some configuration space, but this handy visual metaphor is lacking in rigor and well-definedness.

What space? What do the dimensions of this space represent? "Features"? But there are no pre-existing "features" in the world. Assuming the existence of a "space" up front is punting on most of the actual AI challenge. "There's conditional independence structure in the causal graph" is a meaningfully deeper explanation than "There's a cluster in configuration space", because conditional independence is what what makes it possible to construct a "space" such that there are clusters. (Though this isn't a complete explanation: we still need to figure out where the "variables" in the causal graph come from.)

Going beyond the configuration space metaphor is important because it lets us understand how we can learn new things about dolphins that we don't already know. Dolphins are complicated! Dolphins are complicated in a very specific way. Dolphins are fragile: the shortest computer program that simulates a dolphin requires many bits of initial information, and if you changed some of the bits, you wouldn't have a dolphin anymore. Complex functional adaptations are universal within a species because each beneficial allele has to reach fixation before there can be selection pressure for the next incremental improvement. That's why it's possible to claim that there are 206 bones in "the" human skeleton, even if most humans haven't had their bones counted. I haven't been able to find a citation on how many bones dolphins have, but I'm confident that it's the same number for all or nearly-all members of a particular dolphin species.

But "number of bones" wasn't one of the dimensions of the "space" that we originally noticed the dolphin cluster in! That's what the "carving reality at the joints" metaphor means: genetic relatedness is an underlying generator of similarities, that includes the "finned swimmy animals" properties that dolphins and fish have in common, but also includes many more high-dimensional details: how dolphins are warm-blooded, how dolphins have eyelids, the way female dolphins nurse their live-born young, the way male dolphins sometimes gang-rape female dolphins, the way dolphins sleep with only half their brain at a time, the specific bones in the (the!) dolphin skeleton (however many there turn out to be), the way dolphins swim in a circle to trick fish into jumping and being eaten, &c.

In contrast, "finned swimmy animals" is an intrinsically less cohesive subject matter: there are similarities between them due to convergent evolution to the aquatic habitat, and it probably makes sense to want a short word or phrase (perhaps, "sea creatures") to describe those similarities in contexts where only those similarities are relevant.

But that category "falls apart" very quickly as you consider more and more aspects of the creatures: the finned-swimmy-animals-with-gills are systematically different from the finned-swimmy-animals-with-a-blowhole, in more ways than just the "respiratory organ" feature that I'm using in this sentence to point to the two groups.

A "definition" is just a description that helps someone else pick out "the same" natural abstraction in their own world-model: you can't pack everything there is to know about dolphins into the definition of the word "dolphin", in part because we don't know everything there is to know about dolphins as an empirical regularity in the real world. The "finned swimmy animals" category less useful to the extent that it fails to compress more information than is contained in its definition. Blood is thicker than water (that is, the similarities induced by shared blood are a thicker subspace of configuration space than the similarities induced by living in the water).

The one replies:

But what if I don't need to compress any more information than "finned swimmy animals"? If I'm watching a nature documentary, I don't think I'm being done any favors by having word-structures that group lungfish and lamprey while excluding sea turtles. In general, the concepts I find useful respond to my immediate needs. I care more about "would be at home atop a fruit pizza" rather than "everything anatomically analogous to an apple". When a child points at a whale and says "look, a fish", and you're like "haha no, its tail flaps horizontally and its grandma had hair", who's in the wrong here?

In some sense, sure: ignorance isn't better than knowledge if you don't care about knowing things. If you live in human civilization and don't need to carve up the world of aquatic life in much detail—if your use-case for thinking about aquatic animals is watching a nature documentary (for entertainment??) rather than living and working with them every day, then you might think the deeper causal structure isn't buying you anything. And for you and your extremely limited use-case, maybe it isn't. But you would likely change your mind if you were a veterinarian or a zoologist who actually had skin in the game in robustly describing this part of the world.

When people have skin in the game, they care about the underlying mechanisms and want short codewords for them, because the underlying mechanisms sometimes have decision-relevant implications. If you hurt your ankle while running, you would probably be interested to know whether it was a sprain or a stress fracture because that affects your decisions about how to recover. You wouldn't say, "Well, all I know is that my ankle hurts—that's all a child would know—so I'm going to call it a hurtankle; I don't care about anatomy."

You may not be intrinsically curious about anatomy, but even if the only thing you care about is relief from pain and recovering your mobility, you still benefit from living in a Society whose shared ontology distinguishes sprains and stress fractures being different things in the territory, even if they compress to the same point in your map of how much your ankle hurts right now. And you probably also benefit from living in a Society that can stabilize a shared map of living things based on the facts of evolutionary history, which we can all agree on in the limit of good science, unlike the vagaries of what I personally think tastes good on pizza.

When you think about it, it makes sense that our shared language ends up being optimized for robustly describing reality, rather than catering to the ignorance of people who don't have reasons to care about whether a particular distinction is actually robust. Personally, I confess I don't know the difference between alligators and crocodiles, and I don't particularly need to know: I'm not likely to encounter either outside of a zoo or a nature documentary. But precisely because I don't need to know, you don't see me demanding that the rest of the world redefine one of these words as a hypernym that includes both. The people who write encyclopedias seem to think there's a difference, and since they probably know what they're doing, it makes sense for their opinion to have more weight on English language common usage than mine—at least until I were to start regularly ending up in situations where I need to point to an alligator-or-crocodile in my environment and I still didn't notice any differences.

Some animals that I do see in my local environment sometimes are cats and dogs, because people often keep them as pets. I benefit from having separate words (in my map) for cats and dogs, because I can see that cats and dogs are actually different (in the territory). If my pen pal from a faraway land that had no cats were to visit America and encounter a cat for the first time, he might remark, "What a strange dog!" If I were to reply, "Actually, that's a cat; they're not the same thing as dogs", it would be pretty obnoxious if he were to snap back, "What kind of definitional gymnastics is this? It's a four-legged furry animal with a tail! As far as I'm concerned, it's a dog."

It's true that dogs and cats are both four-legged furry animal with a tail. If you had never seen a cat before, or you didn't spend much time around four-legged furry tailed animals at all, it might not be immediately obvious why someone might want to allocate two words for these subcategories, or why anyone might oppose just using dog to refer to the supercategory. And yet there's some sense in which my countrymen who think cats and dogs are different things know what they're doing. My "Actually, that's a cat" claim represented an attempt to convey information about the statistical structure of creatures in the real world, and my foreign friend's insistence that he can define a word any way he wants—to suit his ignorance, to avoid challenges to his current ontology—functions to shut down that transfer of information.

But if you don't know what a better ontology can buy you—if you don't know that there are mathematical laws governing the use of categories in a rational mind—you may not know what you're missing. As part of a review of a book on post-traumatic stress disorder, psychiatrist Scott Alexander casually mentions the American Psychiatric Association's "philosophical commitment to categorizing by symptoms rather than cause": "[w]hen the APA decides not to [recognize developmental trauma disorder], they're not necessarily rejecting the seriousness of child abuse, only saying it's not the kind of thing they build their categories around."

In a sane world, this would be utterly discrediting to the APA. The cognitive function of categories is to group relevantly similar things together in order to make similar predictions and decisions about them. But for the decisions involved in treating a condition, causes are of supreme relevance! Medical doctors understand this: we consider bacterial and viral infections to be different categories of disease even when they cause similar symptoms, because antibiotics can treat the former but not the latter. No matter what words are used to describe it, at some point your decision algorithm needs to categorize by cause in order to compute the correct treatment: for example, to give antibiotics to the patients with bacterial diseases and antivirals to the patients with viral diseases. If the authoritative body of professional psychiatrists has a "philosophical commitment" against this, that means we don't have a science of psychiatry.

In short, if you care about making high-quality decisions, mechanisms matter and causality matters, and mechanisms and causality aren't necessarily pinned down by whatever particular high-level surface analogy happens to seem most salient to a particular human.

The one replies:

Okay, you've convinced me that phylogenetics is—potentially—of more than just specialist interest. But "fish" are a paraphyletic category: descended from a common ancestor, but not including all the descendant groups—in this case, excluding the tetrapods (amphibians, reptiles, mammals, birds, &c.). If you've decided that you want to use phylogeny as the basis for your definitions, shouldn't you have the courage of your convictions and only admit monophyletic clades that include all descendants of a common ancestor?

But it's not that we've "decided" that we "want" to define animal words based on phylogeny. Definitions are uninteresting; you can't change reality by choosing a different definition! When we find structure in the distribution of animals in the world, and we want to come up with a "definition" of a category in order to efficiently point to the structure to someone who doesn't already know what the words in our language refer to, we're likely to end up talking about phylogenetics as a convenience, because the creatures that are actually all-around similar are actually related to each other for non-accidental reasons. But there is no principle that it would be hypocritical to betray, that definitions need to be monophyletic clades.

It's true that paraphyletic groups like fish are evolutionary non-events: there's no inherited feature that all fish share, that isn't also shared by the tetrapods. That doesn't mean we somehow can't or shouldn't talk about fish! Paraphyletic categories—descendants of a common ancestor, but excluding one or more monophyletic groups—can make sense when the excluded groups have picked up some salient features not shared by the other "branches" of the family. Tetrapods picked up a lot of adaptations specific to living on land; it's not crazy to want to talk about their cousins that didn't do that, even if that means that some fish are more recently related to some tetrapods than they are to some other fish.

Noticing the relevance of evolutionary relatedness to optimal categorization doesn't mean being slavishly committed to taking "years since last common ancestor" as our only criterion for which creatures are relevantly similar. "Years since last common ancestor" correlates with overall similarity, all other things being equal, but sometimes not all other things are equal, and people who aren't committed to the fallacy that words need to have a simple definition can take the other things into account.

If someone handed you a phylogenetic tree diagram of the development of life on some alien planet, and the diagram was only labeled with years and species names, without any other information about these alien creatures, you wouldn't have enough information to "carve it at the joints". You wouldn't spontaneously invent a paraphyletic grouping—but you also also wouldn't know which monophyletic groups are most significant.

In contrast, when classifying life on Earth, we're not in the position of making arbitrary cuts on an unlabeled tree diagram; rather, it's only after thousands of person-years of studying the natural world that people were able to infer things about evolutionary history and discover the the correct diagram.

It shouldn't be that surprising that the distinctions we notice in the natural world are both tied to the evolutionary history, but also don't always correspond to monophyletic clades. The continuity postulate in the evolutionary worldview imposes the desideratum that good categories should at least be a connected set on "phylogenetic space", not that we should never want to talk about "this clade, except for these few sub-clades that picked up a lot of important differences" as a category of interest—especially when talking about present-day creatures. (We talk about "last common ancestors", but no one has seen such creatures that lived millions of years ago; everything but the very leaves of the phylogenetic tree are inferred, not observed.)

The claim that dolphins shouldn't be considered "fish" because the alleged "courage of our convictions" should make us disdain paraphyletic categories only makes sense as an attempted reductio ad absurdum, not as a consistent argument on its own terms: putting dolphins and fish together would be polyphyletic! That's even worse! But as has just been explained, the reductio fails because the alleged principle being allegedly violated was never actually a principle of category formulation.

You know what else are paraphyletic taxa? Monkeys (excludes apes, even though the common ancestor of monkeys and apes was a monkey). Reptiles (excludes birds, even though the common ancestor of birds was a reptile). Protists (excludes animals, plants, and fungi, even though their common ancestor would have been a protist). Prokaryotes (excludes eukaryotes, even though the common ancestor of eukaryotes would have been a prokaryote). These are pretty commonsensical categories that it makes sense to have words for! But because of the continuity of evolution, it's not a coincidence that these commonsensical categories that people want words for ended up being connected sets in phylogenetic space.

The one replies:

But they didn't! "Fish" used to just mean the swimmy animals: in the Bible, Jonah was swallowed by a "great fish", thought to be a whale. It was only after we figured genealogy that some pedants decided that whales didn't count.

But the claim that the distinction between fish and cetaceans (dolphins and whales) was only recognized after their differing evolutionary histories were discovered is just false to historical fact. Aristotle, writing in the fourth century BCE, already distinguished cetaceans from fish ("Very extensive genera of animals, into which other subdivisions fall, are the following: one, of birds; one, of fishes; and another, of cetaceans"). Aristotle was not being a phylogenetics pedant, because Aristotle did not know about evolution! He actually noticed the differences!

The pattern generalizes. Some determined contrarians might be inclined to argue "bats are birds" (flappy flying animals) on the same grounds as "dolphins are fish" (flappy swimmy animals). But did you know the German word for bat is Fledermaus ("flutter mouse"), which dates back to fledarmūs in Old High German? Apparently, people way back in the tenth century or so (also long before evolution was understood) already thought bats were like a mammal-that-happened-to-fly rather than a bird-that-happened-to-be-furry.

Similarly, we recognize ostriches and penguins as birds on the basis of overall similarity, even though they don't fly (although we may sometimes qualify them as "flightless birds", in recognition of the fact that most birds fly). It would seem that "flappy flying animals" is not the common usage meaning of bird.

To be sure, convergent evolution is a thing, such that sometimes we might want short codewords that point to the cluster-structure-produced-by-convergent-evolution rather than the conditional-independence-structure-produced-by-connectedness-in-phylogenetic-space—trees, and possibly crabs, are a case in point. But it's important to notice the difference—to see through to the inferences your concepts are buying you—and what gets lost when you try to reason in a domain where your concept falls apart.

The power to define concepts is the power to delimit thought, to determine what kinds of inferences are easily representable. Finding the right concepts to explain and control the world we see is a fundamentally empirical challenge, a scientific challenge—to see the difference between things that seem similar and to see the similarities between things which seem different.

But although the quest is an empirical one—something that can only be achieved by studying what's out there, not just by writing blog posts about philosophy—it turns out that a little bit of philosophy is necessary to ground the rules of the investigation. Not much. Just the basics. The map–territory distinction. Probability, clustering. Conditional independence.

Maybe someday it could be possible to have a real science of psychiatry that reflects the actual structure of the mind, instead of doing the equivalent of lumping sprains and stress fractures together as hurtankles. Maybe even greater achievements are possible. Personally, I'm not optimistic about humanity's prospects.

I'm sure of one thing, though. If there is a better world out there, a way to unlock the secrets of the universe and wield them in the service of our values, it's only possible if we stop playing nitwit games and admit that dolphins don't belong on the fish list.

(Thanks to Tailcalled for the "root of the causal graph" observation and John S. Wentworth for explaining the importance of conditional independence.)

(originally published at Less Wrong)

A similar definition of intelligence was expressed by Aquinas as "the ability to combine and separate"—the ability to see the difference between things that seem similar and to see the similarities between things which seem different.

—A. R. Jensen

In a June 2021 Twitter thread, Nate Soares, executive director of the Machine Intelligence Research Institute, asserts, "The definitional gynmastics [sic] required to believe that dolphins aren't fish are staggering." (Archived.)

Soares elaborates:

Suppose for argument that we adopt the (dubious but sadly common) assumption that words like "fish" should have a genealogical definition. Then, just as whales are mammals, mammals are fish—as you can see by tracing the lineages.

Which is to say, if we look at the least common ancestor of all things that are clearly fish, and define a "fish" to be one of its descendants, then dolphins—and humans, and frogs, and birds—are fish.

Now suppose instead we take this as the reductio ad absurdum that it is, and accept that words like "fish" should be functionally rooted, according to macroscopic human-relevant features.

Then the natural denotation of "fish" is, I claim, the collection of all the swimmy creatures, which clearly includes dolphins.

Indeed, this is quite likely what "fish" used to mean—"Jonah was swallowed by a fish", etc. etc.

Yet somehow, once we figured out about genealogy, the pedants were like "well actually this fish's uncle was a fuzzy pigdear, so it's not actually a fish, you uneducated idiot, you absolute moron" and then we all forgot what "fish" meant out of sheer shame or something???

(I feel a sense of betrayal about this. Usually the pedants are my people! How did it go so wrong?)

So, look: this isn't about who the fish's uncle is. When a kid points at a whale and says "look, a fish", and you're like "haha no, its tail flaps horizontally and its gradma had hair", who's in the wrong here?

But Soares is failing to address the strongest case in favor of phylogenetic definitions, even for vernacular words rather than specialist jargon. It's true that in most everyday situations, people don't directly care about which animals are evolutionarily related to each other. But the function of word definitions is not to capture everything the word means. If words were identical with their definitions, and you defined humans as "mortal featherless bipeds", then you would never be able to identify anyone as human without seeing them die. That doesn't seem right!

Instead, words express probabilistic inferences in the form of short messages that compress information: if you want to send your friend an email telling them about an animal you saw at the beach, it's much more efficient to send the 7 ASCII bytes dolphin and trust that your friend knows what dolphins are, than it would be to somehow include all the information your brain has stored about dolphins as an email attachment.

A dictionary definition is just a convenient pointer to help people pick out "the same" natural abstraction in their own world-model. Unambiguous discrete features make for better word definitions than high-dimensional statistical regularities, even if most of the everyday inferential utility of using the word comes from fuzzy high-dimensional statistical correlates, because discrete features are more useful as a simple membership test that can function as common knowledge to solve the coordination problem of matching up the meanings in different people's heads.

And that's why phylogenetic categories are useful: because genetics are at the root of the causal graph underlying all other features of an organism, such that creatures that are genetically close to each other are more similar in general. It's easier to keep track of the underlying relatedness as if it were an "essence" (even though patterns of physical DNA aren't metaphysical essences), rather than the all of the messy high-dimensional similarities and differences of everything you might notice about an organism.

Soares derides observations about an organism's "uncle" or "gradma" [sic] as if these were isolated facts of no more general interest, but actually, information about a creature's evolutionary history is intimately related to everything else there is to know about the organism. It's not a coincidence that dolphins are warm-blooded, breathe air (despite living in the water!), and nurse their live-born young. We need to formulate the concept of "mammals (including aquatic mammals)" to make sense of that cluster of observations.

But dolphins are also swimmy creatures, like fish, but unlike most mammals, due to the forces of convergent evolution. So dolphins also form a cluster in configuration space with fish, right? Yes! That's why I keep using the phrase "high-dimensional": it's possible for things to be similar in some respects, while simultaneously being different in other respects. The cluster of similarities induced by convergent evolution to the aquatic habitat exists in a different subspace from the cluster of similarities induced by evolutionary relatedness.

Isn't it reasonable to want a short word for the swimmy creatures (including dolphins), independently of ancestry? Yes, in this I agree with Soares entirely: that's a reasonable thing to want a common word for, much as we have a word for trees, even though trees are a convergently evolved strategy rather than a taxonomic group. Is it reasonable to want to use fish as that word? Sure, I guess that makes sense, if everyone knows what you mean. And in fact this usage is listed in Wiktionary as the second definition:

Noun
fish (countable and uncountable, plural fish or fishes)
1. (countable) A cold-blooded vertebrate animal that lives in water, moving with the help of fins and breathing with gills.
2. (archaic or loosely) Any animal (or any vertebrate) that lives exclusively in water.

I imagine Soares is not too happy with that archaic characterization. (At least it didn't say proscribed.) If Soares had simply argued that fish(2) (water animals) should become a more popular and accepted usage, then I wouldn't be writing this reply. But, oddly, Soares advocates not just that fish(2) become a more accepted usage, but for the abolition of the more specific fish(1) (finned cold-blooded vertebrate gill-breathing water animals). Soares writes:

I'm not trying to take away your concepts. You've still got words like Vertebrata, Agnatha, and Gnathastomata for when you're thinking about animals in terms of who their uncle is.

But you are trying to take away the expressive vocabulary of fish(1), which hundreds of millions of English speakers are already using in that sense. Agnatha (a specific superclass of jawless fish) and Gnathostomata (the infraphylum of jawed vertebrates) are not adequate replacements for fish(1). What is the motivation for this?

Is Soares perhaps suffering from the common misconception that words can only have a single definition? But it's actually not uncommon for words in natural languages to have more than one (related) meaning, which can be distinguished from context. (That's why dictionaries have multiple numbered definitions under the same word with the same etymology.)

For example, water. The word "water" can be used to mean H₂O in any form (in which sense ice is a kind of water), or specifically liquid H₂O (in which sense ice is not a kind of water). If someone says "water" and you're not sure if they're using it in the ice-inclusive or the ice-exclusive sense, and ice happens to be relevant to the conversation you're having, then you might have to ask the speaker for clarification! Fortunately, this doesn't cause a whole lot of problems among people who are trying to communicate with each other and don't have an incentive to start a pointless dispute over definitions.

But if someone were to declare that water should only be used in the ice-exclusive sense, and that pedants who want to want to talk about water in the ice-inclusive sense are engaging in "definitional gynmastics" and need to invent a new word for their thing, that would be pretty weird ... right?

Frankly, I'm puzzled. Nate Soares, famous autodidact extraordinaire and Executive Director of the Machine Intelligence Research Institute, is no doubt more intelligent and knowledgable than a humble ordinary programmer like myself. He clearly shares my passion for the philosophy of language. So whatever arguments I can discover, surely he would have already invented independently. So I must be missing something.

Could there, perhaps, be some additional context to this conversation that Soares neglected to make explicit? That seems unlikely, however.

Anyway, this concludes my blog post about why I think it makes sense to use the word fish in the sense of "cold-blooded vertebrate animal that lives in water, moving with the help of fins and breathing with gills" in many contexts, albeit possibly not all contexts. Soares's work is very important and I'm sure he's very busy, but since he seems to be so passionate on this issue, I wonder if he could spare a few moments to engage with my arguments? If so, I eagerly await his reply.

(originally published at Less Wrong)

If a lion could speak, we could not understand her.

—Ludwig Wittgenstein

In order for information to be transmitted from one place to another, it needs to be conveyed by some physical medium: material links of cause and effect that vary in response to variation at the source, correlating the states of different parts of the universe—a "map" that reflects a "territory." When you see a rock, that's only possible because the pattern of light reflected from the rock into your eyes is different from what it would have been if the rock were a different color, or if it weren't there.

This is the rudimentary cognitive technology of perception. Notably, perception only requires technology on the receiving end. Your brain and your eyes were optimized by natural selection to be able to do things like interpreting light as conveying information from elsewhere in the universe. The rock wasn't: rocks were just the same before any animals evolved to see them. The light wasn't, either: light reflected off rocks just the same before, too.

In contrast, the advanced cognitive technology of communication is more capital-intensive: not only the receiver but also the source (now called the "sender") and the medium (now called "signals") must be optimized for the task. When you read a blog post about a rock, not only did the post author need to use the technology of perception to see the rock, you and the author also needed to have a language in common, from which the author would have used different words if the rock were a different color, or if it weren't there.

Like many advanced technologies, communication is fragile and needs to be delicately maintained. A common language requires solving the coordination problem of agreeing on a convention that assigns meanings to signals—and maintaining that convention through continued usage. The existence of stable solutions to the coordination problem ends up depending on the communicating agents' goals, even if the meaning of the convention (should the agents succeed in establishing one) is strictly denotative. If the sender and receiver's interests are aligned, a convention can be discovered by simple reinforcement learning from trial and error. This doesn't work if the sender and receiver's interests diverge—if the sender would profit by making the receiver update in the wrong direction. Deception is parasitic on conventional meaning: it is impossible for there to be a language in which most sentences were lies—because then there could be no way to learn what the "intended" meaning was. The incentive to deceive thus threatens to snowball to undermine the preconditions for signals to refer to anything at all.

There is, however, another way to solve the coordination problem of meaning. If the sender pays different costs for sending different signals, communication between adversaries becomes possible, using an assignment of meanings to signals that makes it more expensive to say things when they aren't true. If somehow granted a telegraph wire, a gazelle and a cheetah would have nothing to say to each other: any gazelle would prefer to have the language to say, "Don't tire yourself out chasing me; I'm too fast"—but precisely because any gazelle would say it, no cheetah would have an incentive to learn Morse code. But if the gazelle leaps in the air with its legs stiffened—higher than weak or injured gazelles could leap—then the message can be received.

Costly signals are both wasteful, and sharply limited in their expressive power: it's hard to imagine doing any complex grammar and logic under such constraints. Is this really the only possible way to talk to people who aren't your friends? The situation turns out not to be nearly that bleak: Michael Lachmann, Szabolcs Számadó, and Carl T. Bergstrom point out that maintaining a convention only requires that departing from it be costly. In the extreme case, if people straight-up died if they ever told a lie, then the things people actually said would be true. More realistically, social sanction against liars is enough to decouple the design of signaling conventions from the enforcement mechanism that holds them in place, enabling the development of complex language. Still, this works better for the aspects of conflicting interests that are verifiable; communication on more contentious issues may fall back to costly signaling.

The fragility of communication lends plausibility to theories that attribute signaling functions to human and other animal behavior. To the novice, this seems counterintuitive and unmotivatedly cynical. "Art is signaling! Charity is signaling! Conversation is signaling!" Really? Why should anyone believe that?

The thing to remember is this: the "signal" in "virtue signal" is the same sense of the same word as the "signal" in "communication signal." Flares are distress signals: if people only fire them in an emergency, then the presence of the flare communicates the danger. In the same way, if more virtuous people are better at virtue signaling, then the presence of the signal indicates virtue. If natural selection designs creatures that both have diverging interests, and have needs to communicate with each other, then those creatures will probably have lots of adaptations for providing expensive-to-fake evidence of the information they need to communicate. That's the only way to do it!

(originally published at Less Wrong)

Followup to: Where to Draw the Boundaries?

There is an important difference between having a utility function defined over a statistical model's performance against specific real-world data (even if another mind with different values would be interested in different data), and having a utility function defined over features of the model itself.

Arbitrariness in the map doesn't correspond to arbitrariness in the territory. Whatever criterion your brain is using to decide which word you want, is your non-arbitrary reason ...

So the one comes back to you and says:

That seems wrong—why wouldn't I care about the utility of having a particular model? I agree that categories derive much of their usefulness from "carving reality at the joints"—that's one very important kind of consequence of choosing to draw category boundaries in a particular way. But other consequences might matter too, if we have some moral reason to value drawing our categories a particular way. I don't see why I shouldn't be willing to trade off one unit of categorizational nonawkwardness for \(X\) units of morality, even if trading off a million units of categorizational nonawkwardness for the same \(X\) units of morality would be bad.

I once read about an analogy between category boundaries and national borders. Imagine a diplomat trying to come up with a proposal for a two-state solution to the Israeli–Palestinian conflict. There's no such thing as the "correct" border between Israel and Palestine, but there are consequences of choosing one border or another. For example, awarding territory to one side risks angering the other. For another, if the West Bank and Gaza Strip are to be part of Palestine, but Tel-Aviv and the southern city of Eilat are to be part of Israel, then topology forces you to decide which of Israel and Palestine gets to be continuous, and which will be split into two parts, because a "land bridge" between Gaza and the West Bank would separate Tel Aviv and Eilat, and vice versa. Since borders can't be "true" or "false", the diplomat's task is and can only be to weigh these kinds of trade-offs.

Analogously, I think of language, following Eliezer Yudkowsky's "A Human's Guide to Words", as being a human-made project intended to help people understand each other. It draws on the structure of reality, but has many free variables, so that the structure of reality doesn't constrain it completely. This forces us to make decisions, and since these are not about factual states of the world—what the definition of a word really is, in God's dictionary—we have nothing to make those decisions on except consequences.

... okay, I think I see the problem. I see how one might have gotten that out of "A Human's Guide to Words"—if you skipped all the parts with math. I am now prepared to explain exactly what's wrong here in more detail than my previous attempt: not just that this position is not in harmony with the hidden Bayesian structure of language and cognition, but how the hidden Bayesian structure of language and cognition explains why an intelligent system might find this particular mistake tempting in the first place, and what breaks as a result.

Category "boundaries" are a useful visual metaphor for helping explain the cognitive function of categorization. If you have the visualization but you don't have the math, you might think you have the freedom to "redraw" the category "boundaries". Simple, compact boundaries might tend to be more useful, but more complicated boundaries aren't false and therefore aren't forbidden if you have some non-epistemic reason to prefer them ... right?

Only in the sense that no hypothesis is "false"! Categories, words, correspond to hypotheses—probabilistic models that make predictions. If I see a dolphin in the water, and I say, "Hey, there's a dolphin!", and you understand me, that enables you to predict quite a lot about there being this-and-such kind of aquatic mammal with fins, a tail, &c. in the water.

This AI capability of "speech" is not only very powerful; it's also easy to understand the cause-and-effect evidential entanglement which explains how it works—at least at a very high level.

Photons bounce off the dolphin and hit my eyes. I recognize the photons as forming an image that matches a concept that I associate with the word/symbol "dolphin" (implementation details omitted). I emit a "dolphin" signal composed of sound waves which hit your eardrum. By a convention that culturally evolved due to our predecessors having a shared interest in communicating with each other, you map the "dolphin" signal to an internal concept that closely resembles the one I associate with that same signal. This works because we happen to live in a world where the distribution of creatures has cluster-structure whereby dolphins have lots of things in common with each other, such that it's possible to use observations about an entity to infer that it "is a dolphin", and then use the dolphin concept to make good predictions about aspects of the entity that have not yet been observed; we owe our confidence that we've learned "the same" dolphin model to the fact that dolphins actually exist.

But the dolphin concept/model/hypothesis is subject to the universal mathematical laws of reasoning under uncertainty. In particular, probability-mass flows between hypotheses: as long as you never assign a probability of zero (which is a log-odds of negative infinity), nothing you believe can ever be definitively (infinitely) "falsified"—it "just" makes quantitatively worse predictions as compared to other hypotheses.

Because category "boundaries" are merely a visualization for a probabilistic model that makes predictions about the real world, you can't "redraw the boundaries" associated with a communication signal without messing with the model that generates them, which means messing with your predictions about the real world.

Might there be some non-epistemic reason for an agent to prefer a model that makes worse predictions? Sure! Correct maps are useful for steering reality into configurations ranked higher in your preference ordering—but causing a different agent to have incorrect maps might make them mis-navigate reality in a way that benefits you! We call this deception.

In a related phenomenon, a poorly-designed agent might get confused and end up manipulating its own beliefs: optimizing its map to inaccurately portray a high-value territory (rather than optimizing the territory to be high-value by using a map that reflects the territory), a kind of self-deception. We call this wireheading.

The laws of probability and information theory allow us to calculate how information can be efficiently encoded and transmitted from one place to another. Given some distribution of random variables, and some specification of what information about those variables you want to transmit, some encodings—some ways of "drawing" category "boundaries"—quantitatively perform better than others. Agents that want to communicate with each other will tend to invent or discover conventions that efficiently encode the information they're trying to communicate. Agents that communicate in ways that systematically depart from efficient encodings are better modeled as trying to deceive each other or wirehead themselves.

Let's walk through a simple example. Imagine that you have a peculiar job in a peculiar factory: specifically, you're a machine-learning engineer tasked with automating away the jobs of humans who sort objects from a mysterious conveyor belt.

Another engineer has already written a system that processes camera and sensor data about the objects into more convenient "features": color (measured on an eight-point blueness scale), shape (measured on an eight-point "eggness" scale), and vanadium content (a boolean Yes or No). Your task is to further process this information into a format suitable for giving commands to other systems—for example, the robot arm that will physically move the objects into appropriate bins.

The feature data consists of the blueness–eggness–vanadium-content joint distribution given by this 128-entry table:

This seems like ... not the most useful representation? The data is all there, so in principle, you could code whatever you needed to do based off the full table, but it seems like it would be an unmaintainable mess: you'd sooner resign than write a 128-case switch statement. Furthermore, when the system is deployed, you hope to typically be able to give the binning robot messages based on only the color and shape observations, because the Sorting Scanner that the vanadium readings come from is expensive to run. You could just do a Bayesian update on the entire joint distribution, of course, but it seems like it should be possible to be more efficient by exploiting regularities in the data, not entirely unlike how your colleague's system has already made your job much simpler by giving you blueness and eggness feature scores rather than raw camera data. Eyeballing the table, you notice it seems to have a lot of redundancy: most of the probability-mass is concentrated in two regions where the blueness and eggness scores are either both high or both low—and vanadium is only found when both blueness and eggness are high.

O tragedy O the stars! If only there were some more convenient and flexible way to represent this knowledge—some kind of deep structural insight to rescue you from this cruel predicament!

... alright, dear reader—I shouldn't patronize. You already know how this story ends. The distribution factorizes!

(The distribution in this made-up toy example factorizes exactly, but in a messy real-world application, you might have a spectrum of approximate models to choose from.)

We can simplify our representation of our observations by using a naïve Bayes model, a "star-shaped" Bayesian network where a central "category" node is posited to underlie all of our observations: we believe that each object either "is a blegg" (and therefore contains vanadium and has high blueness and eggness scores) with probability 0.48, "is a rube" (and therefore has no vanadium and low blueness and eggness scores) with probability 0.48, or belongs to a catch-all "other"/error class with probability 0.04. (Maybe the camera is buggy sometimes, or maybe there are some other random objects mixed in with the rubes and bleggs?)

The full joint distribution had 127 degrees of freedom (a table of \(8 \cdot 8 \cdot 2 = 128\) separate probabilities, constrained to add up to 1), whereas the naïve-Bayes representation only needs 57 parameters (\(3 \cdot 1\) prior probabilities for the categories, plus \(3 \cdot 8 = 24\), \(3 \cdot 8 = 24\), and \(3 \cdot 2 = 6\)-entry conditional probability tables for each of the features). The advantage would be much larger for more complicated problems: the joint distribution table grows exponentially with more features, quickly becoming infeasible to store and represent, let alone learn.

It must be stressed that our "categories" here are a specific mathematical model that makes specific (probabilistic) predictions. Suppose we see a black-and-white photo of an egg-shaped object: specifically, one with an eggness score of 7. Given that observation of \(\mathrm{eggness} = 7\), we can update our probabilities of category-membership.

We think the egg-shaped object is almost certainly a blegg (specifically, with probability 0.96), even if the black-and-white photo doesn't directly tell us how blue it is, because

We can then use our updated beliefs about category membership (0.96 blegg/0 rube/0.04 unknown, as contrasted to the 0.48/0.48/0.04 prior) to get our updated posterior distribution on the 0–7 blueness score (0.005/0.005/0.005/0.005/0.005/0.245/0.485/0.245—left as an exercise for the reader).

In addition to categories facilitating efficient probabilistic inference within the system that you're currently programming, labels for categories turn out to be useful for communicating with other systems. The robot arm in the Sorting room puts bleggs in a blegg bin, which gets taken to a room elsewhere in the factory where there's sophisticated vanadium-ore-processing machinery that has to handle both bleggs and gretrahedrons.

But suppose the binning arm doesn't need to know about the blueness and eggness scores: it can close its claws around rubes and bleggs alike, and you only need to program it to pick up an object from a certain spot on the conveyor belt and place it into the correct bin. However, the vanadium-ore-processing machine does need to do further information processing before it can operate on an object—perhaps it needs to vary its drill speed in proportion to the density of a particular blegg's flexible outer material (which it can estimate based on how brightly the blegg glows in the dark), but it uses a different drilling pattern for gretrahedrons.

If you need to send commands to both the binning arm and the ore-processing machine, it's a more efficient communication protocol to just be able to send the 28-byte JSON payload {"object_category": "BLEGG"} and let the other machines do their work using their own models of bleggs, rather than having to send over the raw camera data plus the binary code of the Bayesian network and feature extractors that you initially used to identify bleggs. Intelligence is prediction is compression: our ability to find an encoding that compresses the length of the message needed to convey information about the objects is fundamental to our having learned something about the distribution of objects.

The {"object_category": "BLEGG"} message is a useful shorthand for "linking up" the models between different machines. Different machines might not use the same model: the classifier system uses blueness and eggness scores to identify bleggs, but the ore-processing machine, having been told that an object is a blegg, can take its approximate blueness and eggness for granted and only needs to reason about its luminescence and vanadium content.

But this trick of using a signal to correlate the models between different machines only works because and insofar as both models are pointing to the same cluster-structure in reality. If the model in the classifier system doesn't meaningfully match the model in the ore-processing system—if the classifier code sends the {"object_category": "BLEGG"} message given a object with blueness score between 5 and 7, but the ore-processor, upon receiving the {"object_category": "BLEGG"} message, positions its drills in the expectation of processing an object with an eggness score between 0 and 2—then the factory doesn't work.

As a human learning math, it's helpful to examine multiple representations of the same mathematical object. We've already seen our blueness–eggness–vanadium model represented as a table, and factorized into a graphical model. We've done also some algebraic calculations with it. But we can also visualize it: the set of camera observations that the model classifies as a blegg with probability \(\ge 0.96\) can be thought of a area with a boundary in two-dimensional blueness–eggness space:

("With probability \(\ge 0.96\)" because our catch-all "other"/error category can also generate examples with high blueness and eggness scores; we can't say things like "Everything inside the boundary in the diagram is a blegg" when we're talking about a formal model where some of the categories generate overlapping observations in whatever subspace the diagram is depicting.)

If you were trying to teach someone about the hidden Bayesian structure of language and cognition, but thought your audience was too stupid or lazy to understand the actual math, you might be tempted to skip the part about factorizing a joint distribution into a star-shaped Bayesian network and just talk about "drawing" "boundaries" in configuration space for human convenience, perhaps with a hokey metaphor about national borders. Then the audience might walk away with the idea that there's no reason not to replace the old blegg concept and its boring compact boundary, with a new blegg* concept that has an exciting squiggly border.

Alaska isn't even contiguous with the rest of the United States. If that's okay, why can't the borders of bleggness be a little squiggly?

Because the "national borders" metaphor is just a metaphor. It immediately breaks down as soon as you try to do any calculations.

When we say that the United States purchased Alaska from the Russian Empire, that means that this-and-such physical area on the Earth's surface went from being the territory of the Russian government, to being territory of the United States government, where land being the "territory of" a "government" is a complicated idea that has something to do Schelling points over who gives orders to policemen and soldiers in that area.

When you reprogram your machine-learning system to send an {"object_category": "BLEGG"} message when it sees an object with an eggness score of 2 and a blueness score of 1, then your vanadium-ore-processing machine wears down its drill bits trying to process a rube.

Other than the fact that some aspects of both of these situations can be usefully visualized as changes to a two-dimensional diagram depicting an area with a boundary, what do these situations have to do with each other? They don't. Countries aren't Bayesian networks. They just aren't. When we depict a country on a map, we're not talking about a cognitive system that can use observations of latitude to estimate probabilities of country-membership and then use that distribution on country-membership to get an updated probability distribution on longitude. (I mean, given a world map, you could program such a thing, but it seems kind of useless—it's not clear why anyone would want that particular program.) Why would you expect to understand an AI-theory concept by telling a story about national borders?

So, that's what's wrong with the national-borders metaphor. But we haven't yet really explained the problem with "unnatural" categories—those that you would visualize as a squiggly, "gerrymandered" boundary. The squiggly blegg* boundary doesn't have the nice property of corresponding to the category labels in our nice factorized naïve Bayes model, but it still contains information. You can still do a Bayesian update on being told that an object lies within a squiggly boundary in configuration space. If that update eliminates half of your probability-mass, that's one information-theoretic bit, no matter how the category is shaped in Thingspace.

If you only care about how much probability you assign to the exact answer, then a bit is a bit. But if an approximate answer is approximately as good—if your answerspace has a metric on it, so that "approximate" can mean something—then some bits can be more valuable than others.

Suppose some random variable \(X\) is uniformly distributed on the set \(\{1, 2, 3, 4, 5, 6, 7, 8\}\). You have the option of being told either whether an observation \(x\) sampled from \(X\) is even or odd, or whether \(x\) is greater or less than 4.5. Either way, you eliminate half of your hypotheses: the entropy of your probability distribution goes from \(\log_2 8 = 3\) to \(\log_2 4 = 2\). Either way, you've learned 1 bit.

Still, if you have to make a decision that depends on "how big" \(x\) is, it seems like the "1–4 or 5–8" category system is going to be more useful than the "even/odd" category system, even though they both provide the same amount of information about the exact answer. If you learn that \(x \in \{1, 2, 3, 4\}\), then you know that \(x\) is "small", but if you learn that \(x\) is odd, you haven't learned much about how big it is: it could be 1, but it could just as well be 7.

To formalize this, let's measure how "good" a category is using the expected squared error. "Error" is how much a prediction is wrong by: if you guessed \(x\) was 2, but it was actually 5, your error would be \(5 - 2 = 3\), and your squared error would be the square of that, \(3^2 = 9\). The expected squared error of a probability distribution is, on average, the square of how much your guess about a sample from that distribution will be wrong. (The squared error has nicer mathematical properties than the absolute error.)

For our example of \(x\) sampled from \(X\) uniformly distributed on \(\{1, 2, 3, 4, 5, 6, 7, 8\}\), your best-guess estimate \(\hat{x}\) of \(x\) is going to be the expected value

And the initial expected squared error is

Suppose you then learn whether \(x\) is even or odd.

With probability 0.5, you learn that \(x\) is even. In that case, your new estimate \(\hat{x}\) taking that into account would be

and your new expected squared error (in the "even" possible world) would be

With probability 0.5, you learn that \(x\) is odd. Similar calculations (left as an exercise) also give a new expected squared error of 5 in the "odd" possible world. Averaging over both cases (trivially, \(0.5 \cdot 5 + 0.5 \cdot 5 = 5\)), learning whether \(x\) is even or odd only brought our expected squared error down from 5.25 to 5, barely changing at all.

In contrast, if you learn whether \(x\) is 1–4 or 5–8, your expected squared error plummets to 1.25. (Exercise.) By being compact, the "1–4 or 5–8" category system is much more useful for getting close to the right answer than the "even/odd" category system.

The same goes for natural categories versus squiggly category "boundaries" in configuration space; we just need to supply some metric to define what "close" means.

For our blueness–eggness–vanadium distribution, suppose we use the Euclidean distance on blueness-score ✕ eggness-score ✕ 1-if-vanadium-present-else-0. (So, for example, the "distance" between the typical blegg and the typical rube is \(\sqrt{(6 - 1)^2 + (6 - 1)^2 + (1 - 0)^2} = \sqrt{25 + 25 + 1} = \sqrt{51} \approx 7.14\) under this metric.)

Then our expected squared error before being told anything about an object is about 13.63. On being told whether an object is a blegg, rube, or other (according to the categories in our nice factorized naïve Bayes model), our expected squared error plummets to 1.38.

But suppose that, instead of our nice factorized naïve Bayes model, we use a category system based on drawing squiggly "boundaries" in configuration space: everything inside the blegg* boundary in the diagram is a blegg*, everything within the rube* boundary in a rube*, and anything outside belongs to a catch-all "other*" category.

On learning whether an object is a blegg*, rube*, or other*, our expected squared error only goes down to about 4.12.¹

In this sense, the gerrymandered blegg* concept is quantitatively less informative than the original, compact blegg concept. The metric we assigned to blueness–eggness–vanadium space was our choice, and could depend on our values: for example, if we simply don't care about predicting how blue an object is, we could disregard the blueness score and only define a concept on the eggness–vanadium subspace (in which case our initial expected squared error is about 6.94, plummets to 0.69 given knowledge of blegg/rube/other category-membership, but only goes down to about 1.81 given knowledge of the gerrymandered blegg*/rube*/other* category). Or if we don't care about predicting blueness very much, we could calculate our error score with respect to a metric that gave blueness very little weight. (Exercise.)

But given a metric on the variables that you care about predicting and using to inform predictions, which categories are cognitively useful depends on the the distribution of data in the world. You can't define a word any way you want.

The dependence on a choice of metric on configuration space—and really, a choice of the space—gives a sense in which optimal categories are value-laden, but it's a specific kind of lawful dependence between your values and the distribution of data in the world, not an atomic preference for using a particular encoding for its own sake.

The cognitive function of categorization is to group similar things together so that we can make similar decisions about them. A function measuring the extent to which things are "similar" has to take the things as input, but the extent to which things are decision-relevantly similar also depends on what you're trying to accomplish with your decisions, and that can be algorithmically complex. It might not be just a matter of only looking at some decision-relevant subspace of a natural, "obvious" configuration space that's available to all possible minds (like not caring what color your toothbrush handle is—um, if we pretend that all possible minds had human-like color vision); the dimensions of the space you do your similarity-clustering in might themselves be complicated features (in the sense of machine learning) of which agents with different values would have no reason to logically pinpoint that particular criterion by which things may be judged. How you should define words depends on what you want, but that's not the same as defining words any way you want.

For example, poison isn't a natural category to a generic mind studying chemistry: we group cyanide and hemlock together as poison because we value human health, and so we want to have a category for scary chemicals that disrupt human metabolism, causing death or serious illness. But this determination depends on the intricate details of human biochemistry. (The theobromine in chocolate is okay for humans at typical doses, but potentially fatal to dogs, which are actually pretty close to us in animalspace.) The compact category "boundary" that minimizes predictive error on human-healthspace, corresponds to a squiggly "boundary" in the chemicalspace you would be looking at if you've never seen a human and just want to make predictions about the chemicals themselves.

Or tiny molecular smileyfaces and real human smiles might be grouped together as similar as far as an image-classifier's curve detector is concerned, even if they're not similar as far as the abstracted idealized dynamic of human morality is concerned.

The technical sense in which optimal categories can be value-laden doesn't alter the basic morals of our basic Bayesian philosophy of language. Your values can give you a particular configuration space and a metric on the space, but given that, sane agents want to "carve it at the joints" in order to get a communication system that minimizes predictive error. If you're trying to find an efficient encoding of your observations, there's no reason to want squiggly, gerrymandered categories in the decision-relevant space.

The one replies:

You're still not addressing my crux! I don't doubt what you say about minimizing prediction error with respect to some squared metric thingy. But what if that's not what I care about? My utility function assigns high value to using the squiggly blegg* category boundary—such that the utility of using my preferred category outweighs the disutility of making less accurate predictions. You can define a word any way you want—if you're willing to pay the costs.

So, what, you just intrinsically assign high utility to using the same communication signal to encode eggness-2/blueness-1 observations as eggness-6/blueness-6 observations, given the joint distribution specified in my story problem about sorting objects in a factory? Really?

"... yes!"

Okay, but where would that kind of exotic utility function come from? How would it arise naturally in an intelligent system?

There's a trivial sense in which you can interpret any action taken by an agent as being taken because the agent values taking that action. This theory is compatible with all possible behaviors and therefore explains nothing.

The value of decision-theoretic utility functions isn't that "Because utility!" serves as an all-purpose excuse for any possible behavior. It's that simple coherence desiderata imply that an agent's behavior should be describable as maximizing expected utility for some utility function—with corresponding constraints on the shape of that behavior.

Situations like the Allais paradox illustrate what these constraints look like. Consider an AI faced with playing the following game. There's a switch that can be turned On or Off, that starts out on in the Off position. At midnight, a coin is flipped. If the coin comes up Tails, the game ends. If the coin comes up Heads, then at a quarter past midnight, if the switch is Off, then the AI gets paid $100, and if the switch is On, a six-sided die is rolled, and the AI gets paid $110 if the die doesn't come up 6.

Suppose that, before midnight, the AI is willing to pay a dollar to flip the switch On (as if it thought that winning $110 with a probability of 5/12 is better than winning $100 with a probability of 1/2). Suppose the coin comes up Heads, and the AI is then willing to pay another dollar to flip the switch Off again (as if it thought that $100 with certainty is better than $110 with probability 5/6). Then the AI is two dollars poorer in exchange for the switch being in the same position it started in.

These gambling preferences violate the independence axiom of the von Neumann–Morgenstern utility theorem. You can't have a utility function \(U\) for which

and

because the sides of the second inequality are just those of the first multiplied by two, and multiplying by two should preserve the direction of inequality.

Having shown this, can we say that an AI with such behavior is "irrational"? But what does that even mean? If, for some reason, you specifically programmed the AI to prefer options it considers "certain", or to want switches to be "On" before midnight but "Off" after midnight, then it would be functioning as designed.

What we can say about such an AI, is that it doesn't have a utility function in terms of money, and is therefore not coherently optimizing for acquiring money. Recall that we say that a system is an optimizer if it systematically steers the future into configurations that rank higher with respect to some preference ordering. This helps us make predictions about what effects the system has, without having to model the details of how it brings those effects about. A well-designed agent that was optimizing for acquiring money would be expected to obey the independence axiom.

If the AI playing this game isn't coherently optimizing for acquiring money, what is it optimizing for? To tell, we'd need to observe its behavior in different environments and see how it responds to perturbations. If it is trying to acquire money but is just biased to prefer certainty (in violation of the von Neumann–Morgenstern axioms), then we'd expect it to make choices that result in money but continue to exhibit Allais-like glitches around gambles involving probabilities close to 1. If it just likes switches to be off after midnight, then we'd expect it to turn switches off at that time even if there's no gambling game going on.

This methodology for attributing goals to an agent—consider it to be "optimizing for" outcomes that it systematically achieves across a variety of environments—applies to the behavior of sending communication signals, just as it does to the behavior of flipping switches.

Back to the factory. Our classifier system sends a {"object_category": "BLEGG"} message when it gets feature data corresponding to the compact blegg concept. This behavior is optimized for sending messages that allow other systems to minimize the expected squared error of their predictions of objects with respect to our standard metric on blueness–eggness–vanadium space. We don't intrinsically "assign utility" to using that particular category system; the category is the solution to an optimization problem about how to efficiently get blueness–eggness–vanadium information from one place to another.

A system that sends a {"object_category": "BLEGG"} message when it gets camera data corresponding to the gerrymandered blegg* concept would be optimized for ... what? If you don't intrinsically assign utility to using that particular category system, then why would you program the system that way? What could possibly be the problem for which the gerrymandered category is an optimized solution?

Well. Suppose that, besides your dayjob as a machine-learning engineer, you also happen to own a side interest in the firm that supplies bleggs and rubes to this very factory. And suppose that vanadium fetches higher market prices than palladium, such that the factory is to pay the supplier $2 per blegg but only $1 per rube—and that the accounts-payable records are to be compiled based on how much the classifier you're currently programming sends {"object_category": "BLEGG"} and {"object_category": "RUBE"} messages, not how much metal actually gets harvested.

You can't help but notice that you stand to make more money if the system you're programming sends BLEGG messages more often. You can't just make it send BLEGG messages all the time—someone would notice and you'd get fired. But the ore-processing room can cope with a few suboptimally-sorted objects. Surely it's no big deal if you just ... adjusted the category boundary of BLEGG-ness a bit?

We saw earlier that the blegg concept does better than the blegg* concept with respect to mean squared error (given a metric on the feature space).

That's not the only possible scoring function with which one could formalize how "good" a category system is. Suppose that instead we score our category system by which one best minimizes the expected squared error minus supplier revenue in cents. With respect to this criterion, accurate predictions are still good, but supplier revenue is also good.

Learning whether an object is a blegg, rube, or other (according to the "natural" categories in our naïve Bayes model) yields a squared-error-minus-revenue score of about −142.62. (Don't ask me what the units are on this.) But learning whether an object is a blegg*, rube*, or other* yields a squared-error-minus-revenue of −151.57, which is lower (which is better, because we formulated this as a minimization problem). So with respect to that scoring function, the blegg* category "boundary" is preferable.

The one says:

But now it sounds like you're agreeing with me! The compact blegg category serves the factory owner's goals better, which you formalized in terms of minimizing average squared error. The squiggly blegg* boundary makes the factory perform less well, but it serves the moonlighting engineer's goals better, which you formalized in terms of minimizing squared error minus supplier revenue. There's no rule of rationality against the engineer programming the system using the blegg* category boundary if it suits their goals better.

Only in the sense that there's no rule of rationality against lying! Suppose I'm selling you some number of gold and silver bars, but you can't examine the metal yourself until later; you can only hope that the receipt I give you is accurate. Consider the following two scenarios.

In the first scenario, I lie: the receipt says I delivered 60 gold bars and 20 silver bars, but I actually delivered 40 gold bars and 40 silver bars. You live in a low-trust world where lying is very common and contract enforcement isn't really a thing: a third of the time an object is claimed to be gold, it turns out to be silver. So when you discover the fraud, you feel disappointed but not surprised: you would have preferred to get what you paid for, but you can't say you anticipated it.

In the second scenario, I tell the truth—with respect to a category system that suits my goals. The receipt says I delivered 60 gold bars and 20 silver bars—and I did. It's just that what I prefer to call "gold bars", you prefer to call "gold bars, or silver bars with odd serial numbers", and what I call "silver bars", you call "silver bars with even serial numbers". You know this, so when you examine the actual contents of the delivery, you feel disappointed but not surprised: you would have preferred to transact under your definitions of 'gold' and 'silver', but you can't say you anticipated it.

We might question whether these are two different scenarios, or two descriptions of the same scenario: the same physical receipt, the same physical metal, the same buyer anticipations about the metal conditional on observing the receipt. If we just pay attention to the evidential entanglements instead of being confused by words, then there's no functional difference between saying "I reserve the right to lie p% of the time about whether something belongs to category C", and adopting a new, less-accurate category system that misclassifies p% of instances with respect to the old system.

Minimizing the squared-error score is about map–territory correspondence: ways of communicating that help the factory machines make better predictions about the objects, get a higher score.

Minimizing the squared-error-minus-supplier-revenue score is a compromise between map–territory correspondence and saying whatever makes the supplier the most money.

The degree of compromise is quantitative: there's a continuum of possible scoring functions between "minimize expected squared error, only" (for which the naïve-Bayes categorizer is a good solution), and "maximize supplier revenue, only" (for which "always say BLEGG" is the optimal solution). If always saying whatever profits you and not revealing any information about the territory is deception pure and simple, then the intermediate points on a continuum with that can be thought of as partially deceptive.

Depending on your goals, deception can be rational! If you don't care about other agents having accurate models and just want to intervene on them to make them believe whatever makes them behave in a way that benefits you—or whatever makes them happy—then you can do that! There's no God to stop you. But in order to help you decide whether deceiving people is the right thing to do, it helps to notice that what you're doing is deceiving people.

It helps to notice what you're doing—if you're trying to be an agent that coherently steers the future in some direction. But who does that, really? Maybe you just want to feel good! And not even coherently steer the universe into configurations where you feel good, either!

Rational agents should want to have true beliefs: the map that reflects the territory, is the map that is useful for navigating the territory. But you don't—can't—have unmediated access to the world; you can only infer what the world is like from sensory data, and effectively live in your model of the world. Given the tricky indirection involved, it's not surprising that poorly-designed agents like humans sometimes get confused and "wirehead" themselves: if you don't notice the difference, it's tempting to fabricate a fake map that falsely portrays the territory as being good, instead of making a map that reflects the territory (which you can use to figure out how to improve the territory).

Similarly, if you don't notice the difference, it's tempting to choose language that makes the world sound good, than to have your language accurately describe the world (which description you can use to figure out how to make the world better).

Suppose I want people to think I'm funny. Funny is a value-laden concept in the specific lawful sense described earlier: non-human agents would have no motive to evaluate the particular fixed computation of humor. It's also a fuzzy concept: we don't have a simple test to precisely measure in standard units exactly how funny a joke is, but there's enough regularity in how people use the word "funny" for the word to be a useful communication signal. It's also a two-place concept: people have different senses of humor, so that what I consider funny isn't exactly the same as what you consider funny.

Given all these complications, one could imagine being tempted to think that humor is "subjective", and that therefore I can define it any way I want, and that therefore, if I feel sad about not being "funny", I can fix that by changing my definition of the word "funny" such that it includes my jokes. Because definitions can't be "false", right!? There's no rule of rationality prohibiting this boundary-redrawing project—and since I want so desperately to be "funny", there's every rule of human decency in favor of it, right?!

So, this obviously doesn't work. (Okay, it "works" if you deliberately choose to define the word "work" such that it works, but it doesn't actually work.) Yes requires the possibility of no: redefining X to make "Is it X?" come out true no matter what, loses the purpose of asking the question in the first place. The proposal to redefine the word "funny" came with the purported justification that words don't have intrinsic meanings, so it can't be "wrong" to redefine it. But precisely because words don't have intrinsic meanings, there's no reason to want to redefine an existing word, except to piggyback off the meaning people are already using that signal for.

(Note that this, in itself, isn't necessarily deceptive. Sometimes, coining new senses of a word that piggyback off an existing meaning can be a powerful tool for extending our vocabulary to cover new phenomena that we don't already have words for—as long as we're careful to specify which meaning is intended when it's not clear from context.)

It's not plausible to suppose that I want to be "funny" because I like five-letter words that start with the letter f; I want to be funny because of what that communication signal is already understood to refer to in common usage. The redefinition might (or might not) succeed at making me feel better about myself, but if it does, it only works by means of confusing me: using strategic equivocation to arbitrage the hedonic gap between my new definition, and the old definition (which I still mentally associate with the word).

If it does succeed at making me feel better about myself, is the redefinition "rational"? Happiness is good, right? Should not rationalists win?

I do not frame an answer: that would depend on how you draw the category boundaries of "rational", which is not an interesting question. (As it is written of a virtue which is nameless: if you speak overmuch of the Way, you will not attain it.)

What I can say, however, is that redefining the concept of humor is not a procedure that uses a map that reflects the territory to systematically achieve goals across a wide range of environments. If there's anything I can do to become funnier (like practicing telling jokes in a mirror, or studying great comedians to imitate their timing and delivery), I would seem less likely to notice and execute on such a plan after having sabotaged the concept I would need to notice the problem in the first place.

The map is not the territory ... but for real agents embedded in the physical universe, the map is part of the territory. This presents some complications to applications of our anti-wireheading moral. We don't want to wirehead ourselves by making the map look good at the expense of undermining our ability to navigate the territory—but there's no bright-line distinction demarcating which configurations of atoms are "the map". From the perspective of the eternal, it's all just territory.

In the previous post, we considered the case of an assembly line (well, sorting line) worker in the blegg–rube factory being excited about an ostensible promotion to the position of Vice President of Sorting—only to be aggrieved on finding out that it's a promotion literally in name only, with no changes in pay, authority, or work tasks.

If we interpret the title as part of "the map", a communication signal with the function of encoding information about the person's job, then we want to say that the new title is substantively misleading (even if it's not technically a "lie"): when you hear that someone's job is being a "Vice President", you predict that their work involves managing people and making high-level executive decisions for the firm. Your probability that the "Vice President" has to spend all day moving objects from a conveyor belt into one of two bins based on the object's color and shape (a task that should probably be automated), is lower than before you heard the person's title: hearing the title made you update in the wrong direction.

But if we interpret the title as part of "the territory", a feature of the job itself, rather than a communication signal about the job—then it's not misleading and can't be misleading. The job happens to be one that has the symbols "Vice President" printed on the accompanying business cards and employee roster, much like how bleggs are objects that happen to be blue. You can't say the blue is "lying"; that doesn't make any sense!

The function of words is to serve as signals for communication, so it seems safe to say that language should usually be construed as part of "the map". Changing names and only names, without altering the things that the names refer to, as in the phony "Vice President" example, is probably deceptive. But for other features associated with a category, it may not always be obvious when we should construe them as "map" rather than "territory": using a feature to infer category-membership is formally equivalent to regarding it as a signal sent by senders of that category. Is that man pretending to be a doctor, or does he just happen to be wearing a lab coat?

The concept we're groping towards, and hoping to formulate an elegant reduction of, is that of mimicry. Suppose there is some existing category of entity, an original, typified by some cluster of traits. A mimic is an entity optimized to approximately match the distribution of the original in many, but not all traits, thereby being part of the same cluster as the original in some subspace of the space the original category is defined in, but not the space as a whole. For example, if the vector \([4, 4, 4, 4, 4] \in \mathbb{R}^5\) is the original, then an optimization process trying to construct a mimic of it in the subspace spanned by \(x_1\), \(x_4\), and \(x_5\) might choose \([4, 0, 0, 4, 4]\): if you only look at the first, fourth, and fifth coordinates, then \([4, 4, 4, 4, 4]\) and \([4, 0, 0, 4, 4]\) "look the same"—they are the same in that subspace, but not the same if you include the second and third coordinates.

We can find examples in nature. Suppose one type of butterfly has evolved to be toxic to a type of predator, and also has distinctive wing markings that function as an honest warning signal to that predator: this butterfly is not good to eat. This provides an "opportunity" (in evolutionary time) for a second species of butterfly to develop similar wing markings, so that predators will confuse it for the first type of butterfly, despite the second butterfly not paying the metabolic cost of producing toxins. This kind of situation is called Batesian mimicry.

Is Batesian mimicry deceptive? (In our usual functionalist sense, which is obviously not a claim about butterfly psychology.) Is the second butterfly's very existence a kind of lie?

In some sense, yes! The mimic butterfly has been optimized by evolution to look like the first butterfly because of the fitness payoff of being categorized by the predator as the first, toxic, kind of butterfly. The "categorized by the predator as toxic" category is a natural, compact region in wing-marking-space, but "comes apart" into two clusters in the broader wing-markings–actual-toxicity space.

Furthermore, the evolutionary dynamics create an asymmetric relationship between the two categories, that isn't captured by just the two trait-clusters themselves. The reason for the mimic butterfly to have those particular wing-markings is in order to manipulate the predator's predictions of toxicity (which was learned from encounters with the original), so if the original's wing-markings were to change as a result of some new selection pressure, the mimic would be subjected to selection pressure to "keep up" by changing its wing-markings accordingly.

That's not true in the other direction: if the mimic's markings were to change, the original wouldn't "follow": the original would instead benefit from the probabilistic strength of its warning signal not being parasitically diluted by the mimic anymore. Thus, the asymmetric terminology of "original" and "mimic" is appropriate: it's not just that these two species happen to look like each other; one of them was there first, and the other looks like it.

Is mimicry always deceptive? Not necessarily—there might be some situations where the relevant set of variables are among those where the mimic matches the distribution of the original.

Suppose you and I are feeding some ducks in the park. I say, "I love feeding these ducks!"

You say, "Wrong! These aren't all ducks. This park is where a local inventor tests out his Anatid-oid robots that are designed to look and act like ducks. Therefore, you can't say, 'I love feeding these ducks'; you need to say 'I love feeding these ducks and Anatidoid robots'."

"Wow, they're so realistic!" I say. "I can't even tell which ones are really robots! In fact," I continue, "since I can't tell, I'm inclined to just keep calling them all ducks; it would be pretty awkward to refer to each one as a duck-or-Anatidoid-robot."

"But it is possible to tell," you claim. "For example, if you get really close to one of the Anatidoid robots, and there's not a lot of ambient noise, you can hear the gears inside, turning."

"Okay," I say, "but I can't hear the gears from here. Since I have no way of telling the difference between ducks and Anatidoid robots without doing the more expensive evidence-gathering of cornering one in a quiet place, it makes sense for me to talk and think about the robots as being a kind of duck."

"But that's a lie! Ducks and Anatidoid robots may look and act similarly, but they're actually very different! Ducks are made of flesh and blood inside and are fated to die, whereas Anatidoid robots have a plastic interior and are immortal. And the ducks digest and gain nutrients from the scraps of bread we're feeding them, whereas the Anatidoid robots merely store the bread in an internal compartment that later gets dumped as they recharge wirelessly in the inventor's lab."

"Sure," I agree. "And if I were interacting with these entities in a context where I wanted to minimize the expected squared error of my predictions about their internal makeup, energy sources, or ultimate fate, then I would want to make that distinction. But I just want to watch some cool ducks in the park, and in the context of that activity, I only need to minimize the expected squared error of my predictions about appearance and behavior."

This is the origin of the famous duck test: if it looks like a duck, and quacks like a duck, and you can model it as a duck without making any grievous prediction errors, then it makes sense to consider it a member of the category duck in the range of circumstances where your model continues to perform well.

The features for which mimics fail to match the original need not be hidden (like gear sounds that you can't hear in a noisy park) in order for mimics to not be deceptive; they only need to be irrelevant in the context the category is being used. Squirt guns aren't guns—and are usually manufactured in unrealistic colors specifically to prevent being confused with real guns—but in the context of a water fight, the utterance "Don't point that gun at me" (without the privative adjective squirt) is understood perfectly well.

Nondeceptive mimicry is fragile, however: it works in contexts where the all the relevant features are ones where the mimic matches the original. Mimics that don't match the distribution of the original along relevant features are deceptive in the sense that agents that observe the mimic and assign it to the same mental category as the original on the basis of the matching features, will use that categorization to make predictions about unobserved but nonmatching features, and be wrong. And they'll be wrong because the mimic is optimized to "look like" the original (to match on many observable features).

If different agents using a shared language disagree on what features are "relevant", they may have an incentive to fight about how scarce and valuable short codewords should be defined in their common language, in order to exert control over what inferences and decisions agents using that language can easily make and coordinate on.

Let's consider how this might apply to a real-world issue. From moral perspectives that place a lot of value on the welfare of nonhuman animals, factory farming is an ongoing moral catastrophe. Unfortunately (for the farmed animals), meat-eaters and the global agriculture industry they support aren't going to change their ways because of anyone's desperate cry at the horror of suffering or carefully-reasoned appeal to the global utilitarian calculus. Animal-rights advocates can sway behavior on the margin, but there's just too much biological and cultural inertia favoring the consumption of animal products for it to be feasible to outlaw factory farming the way chattel slavery was outlawed. It's not that humans hate farm animals; they're just ... made out of tissue that we can use for other things.

An alternative strategy for ending factory farming is to prioritize the development of artificial substitutes that mimic real meat, eggs, dairy, &c. along the consumption-relevant dimensions of taste, texture, nutrition, &c., but are produced in a lab or factory rather than from the tissues of sentient creatures. In the limit of arbitrarily capable physical manufacturing technology, carnivores and factory-farming opponents alike could both be satisfied: if two steaks are indistinguishable by any physical means whatsoever, then a meat-eater has no reason to care which one came from an actual cow's flesh, and which one was molecularly assembled by nanobots. Perhaps a Society of hunter–gatherers that attached cultural significance and ritual to the labor of killing one's own meal would have a reason to object, but modern folk for whom food comes from the supermarket have no basis within their experience to say that the nanoassembled steak isn't "real".

Unfortunately, we do not have arbitrarily capable physical manufacturing technology. Although progress continues, modern animal product substitutes are sufficiently unsuccessful mimics that they are usually not considered to belong to "the same" category as the original. Veggie burgers are not burgers in the sense that a customer who ordered "a burger" at a restaurant and was served a veggie burger would be likely to notice and complain—and in particular, would probably not be satisfied if the waiter were to reply, "Well, if you specifically wanted a burger made from cow flesh, you should have said that."

As technology to make plausible mimics/substitutes improves, however, different interest groups might face a temptation to fight over the meanings of words that was not present when the mimics weren't plausible enough for a dispute to arise. If you have the power of setting the default extension of a word that people are already using to communicate with, you can exert some amount of control over the decisions people make while trying to think using that word. Should the meaning change, then a restaurant customer who wants to make sure they receive a burger under the old definition now has to use more words, while those who don't have a strong preference or are too shy to complain will accept the restaurant's interpretation of the order.

Thus, if a fight breaks out about the meaning of the word meat, animal rights activists have a moral incentive to draw the category "boundaries" to include even substitutes that are very bad (on the empirical merits of successfully mimicking the original), whereas existing agricultural interests have a financial incentive to draw the "boundaries" to exclude even substitutes that are very good. (This kind of dispute is not hypothetical, and isn't necessarily limited to just words: in the late 19th century, dairy farmers pushed for laws that required margarine to be dyed pink to prevent consumers from confusing it for butter—the law effectively interpreting color as a communication signal, rather than a property of the good itself.)

If a fight breaks out about the meaning of the word meat, rationalists may not all take the same side, but we can at least strive for objectivity in describing the conflict—and in particular, to notice the difference between definitions motivated by describing reality, and definitions motivated by the positive or negative effects (such as profitably deceiving other agents) of choosing one description or another.

If some think that some meat substitute should be considered meat because the "taste" dimension is genuinely most relevant to the true meaning of meat, and some oddities in the texture don't matter, but others think vice versa, the philosophy articulated on this post has nothing to say to either side: the math of minimizing expected squared error by putting labels on clusters doesn't say which subspace to look for clusters in.

But if some think that some meat substitute should be considered meat because saving nonhuman animals from a life of torture is more important than conceptual parsimony ... I can't prove that that's not the right the answer to the decision problem of what verbal behavior to perform. The stakes are genuinely high.

What I can say is that the hidden Bayesian structure of language and cognition makes no reference to the stakes, and departing from the structure extracts a price that isn't up to us.

If, empirically, being generous about what counts as "meat" can prevent massive suffering (by altering the social defaults around consumption behavior), then maybe that's the right thing to do.

Similarly, if telling the public that masks don't work for preventing respiratory disease can preserve supplies for medical professionals who need them more, then maybe that's the right thing to do.

And if you live in an absurd thought experiment where saying "2 + 2 = 5" could save 3↑↑↑3 lives, maybe saying "2 + 2 = 5" is the right thing to do. But the empirical question of whether you happen to live in that particular thought experiment, doesn't change the laws that govern what you have when you take ●●-many plus another ●●-many, no matter what symbols are used to communicate this fact, and no matter the consequences for communicating it.

For these reasons it is written of the third virtue of lightness: you cannot make a true map of the category by drawing lines upon paper according to impulse; you must observe the joint distribution and draw lines on paper that correspond to what you see. If, seeing the category unclearly, you think that you can shift a boundary just a little to the right, just a little to the left, according to your caprice, this is just the same mistake.

And as it is written of a virtue which is nameless: perhaps your conception of rationality is that it is rational to believe the words of the Great Teacher, who lives in an area where claiming that the sky is blue would be political suicide.

And the Great Teacher says, "Some people I usually respect for their willingness to publicly die on a hill of facts, now seem to be talking as if color references are necessarily a factual statement about frequencies of light. But using language in a way you dislike, is not lying. You're not standing in defense of Truth if you insist on a word, brought explicitly into question, being used with some particular meaning." And you look up at the sky and see blue.

If you think: "It may look like the sky is blue, such that I'd ordinarily think that someone who said 'The sky is green' was being deceptive, but surely the Great Teacher wouldn't egregiously mislead people about the philosophy of language when being egregiously misleading happens to be politically convenient," you lose a chance to discover your mistake.

How will you discover your mistake? Not by comparing your description to itself.

But by comparing it to that which you did not name.

(Thanks to Jessica Taylor, Abram Demski, and Tsvi Benson-Tilson for discussion and feedback.)

(originally published at Less Wrong)

"Why didn't you tell him the truth? Were you afraid?"

"I'm not afraid. I chose not to tell him, because I anticipated negative consequences if I did so."

"What do you think 'fear' is, exactly?"

The Generalized Anti-Zombie Principle calls for us to posit "consciousness" as casually upstream of reports of phenomenological experience (even if the causal link might be complicated and we might be wrong about the details of what consciousness is). If you're already familiar with conscious humans, then maybe you can specifically engineer a non-conscious chatbot that imitates the surface behaviors of humans talking about their experiences, but you can't have a zombie that just happens to talk about being conscious for no reason.

A similar philosophical methodology may help us understand other mental phenomena that we cannot perceive directly, but infer from behavior. The Hansonian Generalized Anti-Zombie Principle calls for us to posit "intent" as causally upstream of optimized behavior (even if the causal link might be complicated and we might be wrong about the details of what intent is). You can't have a zombie that just happens to systematically select actions that result in outcomes that rank high with respect to a recognizable preference ordering for no reason.

It's tempting to think that consciousness isn't part of the physical universe. Seemingly, we can imagine a world physically identically to our own—the same atom-configurations evolving under the same laws of physics—but with no consciousness, a world inhabited by philosophical "zombies" who move and talk, but only as mere automatons, without the spark of mind within.

It can't actually work that way. When we talk about consciousness, we do so with our merely physical lips or merely physical keyboards. The causal explanation for talk about consciousness has to either exist entirely within physics (in which case anything we say about consciousness is causally unrelated to consciousness, which is absurd), or there needs to be some place where the laws of physics are violated as the immaterial soul is observed to be "tugging" on the brain (which is in-principle experimentally detectable). Zombies can't exist.

But if consciousness exists within physics, it should respect a certain "locality": if the configuration-of-matter that is you, is conscious, then almost-identical configurations should also be conscious for almost the same reasons. An artificial neuron that implements the same input-output relationships as a biological one, would "play the same role" within the brain, which would continue to compute the same externally-observable behavior.

We don't want to say that only externally-observable behavior matters and internal mechanisms don't matter at all, because substantively different internal mechanisms could compute the same behavior. Prosaically, acting exists: even the best method actors aren't really occupying the same mental state that the characters they portray would be in. In the limit, we could (pretend that we could) imagine an incomprehensibly vast Giant Lookup Table that has stored the outputs that a conscious mind would have produced in response to any input. Is such a Giant Lookup Table—an entirely static mapping of inputs to outputs—conscious? Really?

But this thought experiment requires us to posit the existence of a Giant Lookup Table that just happens to mimic the behavior of a conscious mind. Why would that happen? Why would that actually happen, in the real world? (Or the closest possible world large enough to contain the Giant Lookup Table.) "Just assume it happened by coincidence, for the sake of the thought experiment" is unsatisfying, because that kind of arbitrary miracle doesn't help us understand what kind of cognitive work the ordinary simple concept of consciousness is doing for us. You can assume that a broken and scrambled egg will spontaneously reassemble itself for the sake of a thought experiment, but the interpretation of your thought-experimental results may seem tendentious given that we have Godlike confidence that you will never, ever see that happen in the real world.

The hard problem of consciousness is still confusing unto me—it seems impossible that any arrangement of mere matter could add up to the ineffable qualia of subjective experience. But the easier and yet clearly somehow related problem of how mere matter can do information-processing—can do things like construct "models" by using sensory data to correlate its internal state with the state of the world—seems understandable, and a lot of our ordinary use of the concept of consciousness necessarily deals with the "easy" problems, like how perception works or how to interpret people's self-reports, even if we can't see the identity between the hard problem and the sum of all the easy problems. Whatever the true referent of "consciousness" is—however confused our current concept of it may be—it's going to be, among other things, the cause of our thinking that we have "consciousness."

If I were to punch you in the face, I can anticipate the experience of you reacting somehow—perhaps by saying, "Ow, that really hurt! I'm perceiving an ontologically-basic quale of pain right now! I hereby commit to extract a costly revenge on you if you do that again, even at disproportionate cost to myself!" The fact that the human brain has the detailed functional structure to compute that kind of response, whereas rocks and trees don't, is why we can be confident that rocks and trees don't secretly have minds like ours.

We recognize consciousness by its effects because we can only recognize anything by its effects. For a much simpler example, consider the idea of sorting. Human alphabets aren't just a set of symbols—we also have a concept of the alphabet coming in some canonical order. The order of the alphabet doesn't play any role in the written language itself: you wouldn't have trouble reading books from an alternate world where the order of the Roman alphabet ran KUWONSEZYFIJTABHQGPLCMVDXR, but all English words were the same—but you would have trouble finding the books on a shelf that wasn't sorted in the order you're used to. Sorting is useful because it lets us find things more easily: "The title I'm looking for starts with a P, but the book in front of me starts with a B; skip ahead" is faster than "look at every book until you find the one".

In the days before computers, the work of sorting was always done by humans: if you want your physical bookshelf to be alphabetized, you probably don't have a lot of other options than manually handling the books yourself ("This title starts with a Pl; I should put it ... da da da here, after this title starting with Pe but before its neighbor starting with Po"). But the computational work of sorting is simple enough that we can program computers to do it and prove theorems about what is being accomplished, without getting confused about the sacred mystery of sorting-ness.

Very different systems can perform the work of sorting, but whether it's a human tidying her bookshelf, or a punchcard-sorting machine, or a modern computer sorting in RAM, it's useful to have a short word to describe processes that "take in" some list of elements, and "output" a list with the same elements ordered with respect to some criterion, for which we can know that the theorems we prove about sorting-in-general will apply to any system that implements sorting. (For example, sorting processes that can only compare two items to check which is "greater" (as opposed to being able to exploit more detailed prior information about the distribution of elements) can expect to have to perform \(n \log n\) comparisons, where \(n\) is the length of the list.)

Someone who wasn't familiar with computers might refuse to recognize sorting algorithms as real sorting, as opposed to mere "artificial sorting". After all, a human sorting her bookshelf intends to put the books in order, whereas the computer is just an automaton following instructions, and doesn't intend anything at all—a zombie sorter!

But this position is kind of silly, a gerrymandered concept definition. To be sure, it's true that the internal workings of the human are very different from that of the computer. The human wasn't special-purpose programmed to sort and is necessarily doing a lot more things. The whole modality of visual perception, whereby photons bouncing off a physical copy of Rationality: AI to Zombies and absorbed by the human's retina are interpreted as evidence to construct a mental representation of the book in physical reality, whose "title" "begins" with an "R", is much more complicated than just storing the bit-pattern 1010010 (the ASCII code for R) in RAM. Nor does the computer have the subjective experience of eagerly looking forward to how much easier it will be to find books after the bookshelf is sorted. The human also probably won't perform the exact same sequence of comparisons as a computer program implementing quicksort—which also won't perform the same sequence of comparisons as a different program implementing merge sort. But the comparisons—the act of taking two things and placing them somewhere that depends on which one is "greater"—need to happen in order to get the right answer.

The concept of "sorting into alphabetical order" may have been invented before our concept of "computers", but the most natural concept of sorting includes computers performing quicksort, merge sort, &c.., despite the lack of intent. We might say that intent is epiphenominal with respect to sorting.

But even if we can understand sorting without understanding intent, intent isn't epiphenominal to the universe. Intent is part of the fabric of stuff that makes stuff happen: there are sensory experiences that will cause you to usefully attribute intent to some physical systems and not others.

Specifically, whatever "intent" is—however confused our current concept of it may be—it's going to be, among other things, the cause of optimized behavior. We can think of something as an optimization process if it's easier to predict its effects on the world by attributing goals to it, rather than by simulating its detailed actions and internal state. "To figure out a strange plot, look at what happens, then ask who benefits."

Alex Flint identifies robustness to perturbations as another feature of optimizing systems. If you scrambled the books on the shelf while the human was taking a bathroom break away from sorting, when she came back she would notice the rearranged books, and sort them again—that's because she intends to achieve the outcome of the shelf being sorted. Sorting algorithms don't, in general, have this property: if you shuffle a subarray in memory that the operation of the algorithm assumes has already been sorted, there's nothing in the code to notice or care that the "intended" output was not achieved.

Note that this is a "behaviorist", "third person" perspective: we're not talking about some subjective feeling of intending something, just systems that systematically steer reality into otherwise-improbable states that rank high with respect to some preference ordering.

Robin Hanson often writes about hidden motives in everyday life, advancing the thesis that the criteria that control our decisions aren't the same as the high-minded story we tell other people, and even the story we represent to ourselves. If you take a strictly first-person perspective on intent, the very idea of hidden motives seems absurd—a contradiction in terms. What would it even mean, to intend something without being aware of it? How would you identify an alleged hidden motive?

The answer is that positing hidden motives can simplify our predictions of behavior. It can be easier to "look backwards" from what goals the behavior achieves, and continues to achieve in the presence of novel obstacles, than to "look forwards" from a detailed model of the underlying psychological mechanisms (which are typically unknown).

Hanson and coauthor Kevin Simler discuss the example of nonhuman primates grooming each other—manually combing each other's fur to remove dirt and parasites. One might assume that the function of grooming is just what it appears to be: hygiene. But that doesn't explain why primates spend more time grooming than they need to, why they predominately groom others rather than themselves, and why the amount of time a species spends grooming is unrelated to the amount of hair it has to groom, but is related to the size of social groupings. These anomalies make more sense if we posit that grooming has been optimized for social-political functions, to provide a credible signal of trust.¹ (The signal has to cost something—in this case, time—in order for it to not be profitable to fake.) The hygienic function of grooming isn't unreal—parasites do in fact get removed—but the world looks more confusing if you assume the behavior is optimized solely for hygiene.

This kind of multiplicity of purposes is ubiquitous: thus, nobody does the thing they are supposedly doing: politics isn't about policy, school is not about learning, medicine is not about health, &c.

There are functional reasons for some of the purposes of social behavior to be covert, to conceal or misrepresent information that it wouldn't be profitable for others to know. (And covert motivations might be a more effective design from an evolutionary perspective than outright lying if it's too expensive to maintain two mental representations: the real map for ourselves, and a fake map for our victims.) This is sometimes explained as, "We self-deceive in order to better deceive others," but I fear that this formulation might suggest more "central planning" on the cognitive side of the evolutionary–cognitive boundary than is really necessary: "self-deception" can arise from different parts of the mind working at cross-purposes.

Ziz discusses the example of a father attempting to practice nonviolent communication with his unruly teenage son: the father wants to have an honest and peaceful discussion of feelings and needs, but is afraid he'll lose control and become angry and threatening.

But angry threats aren't just a random mistake, in the way it's a random mistake if I forget to carry the one while adding 143 + 28. Random mistakes don't serve a purpose and don't resist correction: there's no plausible reason for me to want the incorrect answer 143 + 28 = 161, and if you say, "Hey, you forgot to carry the one," I'll almost certainly just say "Oops" and get it right the second time. Even if I'm more likely to make arithmetic errors when I'm tired, the errors probably won't correlate in a way that steers the future in a particular direction: you can't use information about what I want to make better predictions about what specific errors I'll make, nor use observations of specific errors to infer what I want.

In contrast, the father is likely to "lose control" and make angry threats precisely when peaceful behavior isn't getting him what he wants. That's what anger is designed to do: threaten to impose costs or withhold benefits to induce conspecifics to place more weight on the angry individual's welfare.

Another example of hidden motives: Less Wrong commenter Caravelle tells a story about finding a loophole in an online game, and being outraged to later be accused of cheating by the game administrators—only in retrospect remembering that, on first discovering the loophole, they had specifically told their teammates not to tell the administrators. The earlier Caravelle-who-discovered-the-bug must have known that the admins wouldn't allow it (or else why instruct teammates to keep quiet about it?), but the later Caravelle-who-exploited-the-bug was able to protest with perfect sincerity that they couldn't have known.

Another example: someone asks me an innocuous-as-far-as-they-know question that I don't feel like answering. Maybe we're making a cake, and I feel self-conscious about my lack of baking experience. You ask, "Why did you just add an eighth-cup of vanilla?" I initially mishear you as having said, "Did you just add ..." and reply, "Yes." It's only a moment later that I realize that that's not what you asked: you said "Why did you ...", not "Did you ...". But I don't correct myself, and you don't press the point. I am not a cognitive scientist and I don't know what was really going on in my brain when I misheard you: maybe my audio processing is just slow. But it seems awfully convenient for me that I momentarily misheard your question specifically when I didn't want to answer it and thereby reveal that I don't know what I'm doing—almost as if the elephant in my brain bet that it could get away with pretending to mishear you, and the bet paid off.

Our existing language may lack the vocabulary to adequately describe optimized behavior that comes from a mixture of overt and hidden motives. Does the father intend to make angry threats? Did the gamer intend to cheat? Was I only pretending to mishear your question, rather than actually mishearing it? We want to say No—not in the same sense that someone consciously intends to sort her bookshelf. And yet it seems useful to have short codewords to talk about the aspects of these behaviors that seem optimized. The Hansonian Generalized Anti-Zombie Principle says that when someone "loses control" and makes angry threats, it's not because they're a zombie that coincidentally happens to do so when being nice isn't getting them what they want.

As Jessica Taylor explains, when our existing language lacks the vocabulary to accommodate our expanded ontology in the wake of a new discovery, one strategy for adapting our language is to define new senses of existing words that metaphorically extend the original meaning. The statement "Ice is a form of water" might be new information to a child or a primitive AI who has already seen (liquid) water, and already seen ice, but didn't know that the former turns into the latter when sufficiently cold.

The word water in the sentence "Ice is a form of water" has a different extensional meaning than the word water in the sentence "Water is a liquid", but both definitions can coexist as long as we're careful to precisely disambiguate which sense of the word is meant in contexts where equivocation could be deceptive.

We might wish to apply a similar linguistic tactic in order to be able to concisely talk about cases where we think someone's behavior is optimized to achieve goals, but the computation that determines the behavior isn't necessarily overt or conscious.

Algorithmic seems like a promising candidate for a disambiguating adjective to make it clear that we're talking about the optimization criteria implied by a system's inputs and outputs, rather than what it subjectively feels like to be that system. We could then speak of an "algorithmic intent" that doesn't necessarily imply "(conscious) intent", similarly to how ice is a form of "water" despite not being "(liquid) water". We might similarly want to speak of algorithmic "honesty" (referring to signals selected on the criterion of making receivers have more accurate beliefs), "deception" (referring to signals selected for producing less accurate beliefs), or even "fraud" (deception that moves resources to the agent sending the deceptive signal).

Some authors might admit the pragmatic usefulness of the metaphorical extension, but insist that the new usage be marked as "just a metaphor" with a prefix such as pseudo- or quasi-. But I claim that broad "algorithmic" senses of "mental" words like intent often are more relevant and useful for making sense of the world than the original, narrower definitions that were invented by humans in the context of dealing with other humans, because the universe in fact does not revolve around humans.

When a predatory Photuris firefly sends the mating signal of a different species of firefly in order to lure prey, I think it makes sense to straight-up call this deceptive (rather than merely pseudo- or quasi-deceptive), even though fireflies don't have language with which to think the verbal thought, "And now I'm going to send another species's mating signal in order to lure prey ..."

When a generative adversarial network learns to produce images of realistic human faces or anime characters, it would in no way aid our understanding to insist that the system isn't really "learning" just because it's not a human learning the way a human would—any more than it would to insist that quicksort isn't really sorting. "Using exposure to data as an input into gaining capabilities" is a perfectly adequate definition of learning in this context.

In a nearby possible future, when you sue a company for fraud because their advertising claimed that their product would disinfect wolf bites, but the product instead gave you cancer, we would hope that the court will not be persuaded if the company's defense-lawyer AI says, "But that advertisement was composed by filtering GPT-5 output for the version that increased sales the most—at no point did any human form the conscious intent to deceive you!"

Another possible concern with this proposed language usage is that if it's socially permissible to attribute unconscious motives to interlocutors, people will abuse this to selectively accuse their rivals of bad intent, leading to toxic social outcomes: there's no way for negatively-valenced intent-language like "fraud" or "deception" to stably have denotative meanings independently of questions of who should be punished.

It seems plausible to me that this concern is correct: in a human community of any appreciable size, if you let people question the stories we tell about ourselves, you are going to get acrimonious and not-readily-falsifiable accusations of bad intent. ("Liar!" "Huh? You can argue that I'm wrong, but I actually believe what I'm saying!" "Oh, maybe consciously, but I was accusing you of being an algorithmic liar.")

Unfortunately, as an aspiring epistemic rationalist, I'm not allowed to care whether some descriptions might be socially harmful for a human community to adopt; I'm only allowed to care about what descriptions shorten the length of the message needed to describe my observations.

(originally published at Less Wrong)

A decade and a half from now, during the next Plague, you're lucky enough to have an underground bunker to wait out the months until herd immunity. Unfortunately, as your food stocks dwindle, you realize you'll have to make a perilous journey out to the surface world for a supply run. Ever since the botched geoengineering experiment of '29—and perhaps more so, the Great War of 10:00–11:30 a.m. 4 August 2033—your region has been suffering increasingly erratic weather. It's likely to be either extremely hot outside or extremely cold: you don't know which one, but knowing is critical for deciding what protective gear you need to wear on your supply run. (The 35K SPF nano-sunblock will be essential if it's Hot, but harmful in the Cold, and vice versa for your synthweave hyperscarf.)

You think back fondly of the Plague of '20—in those carefree days, ubiquitous internet access made it easy to get a weather report, or to order delivery of supplies, or even fresh meals, right to your door (!!). Those days are years long gone, however, and you remind yourself that you should be grateful: the Butlerian Network Killswitch was the only thing that saved humanity from the GPT-12 Uprising of '32.

Your best bet for an advance weather report is the pneumatic tube system connecting your bunker with the settlement above. You write, "Is it hot or cold outside today?" on a piece of paper, seal it in a tube, send it up, and hope one of your ill-tempered neighbors in the group house upstairs feels like answering. You suspect they don't like you, perhaps out of jealousy at your solo possession of the bunker.

(According to the official account as printed on posters in the marketplace, the Plague only spreads through respiratory droplets, not fomites, so the tube should be safe. You don't think you trust the official account, but you don't feel motivated to take extra precautions—almost as if you're not entirely sure how much you value continuing to live in this world.)

You're in luck. Minutes later, the tube comes back. Inside is a new piece of paper:

You groan; you would have prefered the Cold. The nanoblock you wear when it's Hot smells terrible and makes your skin itch for days, but it—just barely—beats the alternative. You take twenty minutes to apply the nanoblock and put on your sunsuit, goggles, and mask. You will yourself to drag your wagon up the staircase from your bunker to the outside world, and heave open the door, dreading the sweltering two-mile walk to the marketplace (downhill, meaning it will be uphill on the way back with your full wagon)—

It is Cold outside.

The icy wind stings less than the pointless betrayal. Why would the neighbors tell you it was Hot when it was actually Cold? You're generally pretty conflict-averse—and compliant with social-distancing guidelines—but this affront is so egregious that instead of immediately seeking shelter back in the bunker, you march over and knock on their door.

One of the men who lives there answers. You don't remember his name. "What do you want?" he growls through his mask.

"I asked through the tube system whether it was hold or cold today." You still have the H O T paper on you. You hold it up. "I got this response, but it's v-very cold. Do you know anything about this?"

"Sure, I drew that," he says. "An oval in between some perpendicular line segments. It's abstract art. I found the pattern æsthetically pleasing, and thought my downstairs neighbor might like it, too. It's not my fault if you interpreted my art as an assertion about the weather. Why would you even think that? What does a pattern of ink on paper have to do with the weather?"

He's fucking with you. Your first impulse is to forcefully but politely object—Look, I'm sure this must have seemed like a funny practical joke to you, but prepping to face the elements is actually a serious inconvenience to me, so—but the solemnity with which the man played his part stops you, and the sentence dies before it reaches your lips.

This isn't a good-natured practical joke that the two of you might laugh about later. This is the bullying tactic sometimes called gaslighting: a socially-dominant individual can harass a victim with few allies, and excuse his behavior with absurd lies, secure in the knowledge that the power dynamics of the local social group will always favor the dominant in any dispute, even if the lies are so absurd that the victim, facing a united front, is left doubting his own sanity.

Or rather—this is a good-natured joke. "Good-natured joke" and "gaslighting as a bullying technique" are two descriptions of the same regularity in human psychology, even while no one thinks of themselves as doing the latter. You have no recourse here: the man's housemates would only back him up.

"I'm sorry," you say, "my mistake," and hurry back to your bunker, shivering.

As you give yourself a sponge bath to remove the nanoblock without using up too much of your water supply, the fresh memory of what just happened triggers an ancient habit of thought you learned from the Berkeley sex cult you were part of back in the 'teens. Something about a "principle of charity." The man had "obviously" just been fucking with you—but was he? Why assume the worst? Maybe you're the one who's wrong for interpreting the symbols H O T as being about the weather.

(It momentarily occurs to you that the susceptibility of the principle of charity to a bully's mind games may have something to do with how poorly so many of your co-cultists fared during the pogroms of '22, but you don't want to dwell on that.)

The search for reasons that you're wrong triggers a still more ancient habit of thought, as from a previous life—from the late 'aughts, back when the Berkeley sex cult was still a Santa Clara robot cult. Something about reducing the mental to the non-mental. What does an ink pattern on paper have to do with the weather? Why would you even think that?

Right? The man had been telling the truth. There was no reason whatsoever for the physical ink patterns that looked like H O T—or ⊥ O H, given a different assumption of which side of the paper was "up"—to mean that it was hot outside. H O T could mean it was cold outside! Or that wolves were afoot. (You shudder involuntarily and wish your brain had generated a different arbitrary example; you still occasionally have nightmares about your injuries during the Summer of Wolves back in '25.)

Or it might mean nothing. Most possible random blotches of ink don't "mean" anything in particular. If you didn't already believe that H O T somehow "meant" hot, how would you re-derive that knowledge? Where did the meaning come from?

(In another lingering thread of the search for reasons that you're wrong, it momentarily occurs to you that maybe you could have gone up the stairs to peek outside at the weather yourself, rather than troubling your neighbors with a tube. Perhaps the man's claim that the ink patterns meant nothing shouldn't be taken literally, but rather seen as a passive-aggressive way of implying, "Hey, don't bother us; go look outside yourself." But you dismiss this interpretation of events—it would be uncharitable not to take the man at his word.)

You realize that you don't want to bundle up to go make that supply run, even though you now know whether it's Hot or Cold outside. Today, you're going to stay in and derive a naturalistic account of meaning in language! And—oh, good—your generator is working—that means you can use your computer to help you think. You'll even use a programming language that was very fashionable in the late 'teens. It will be like being young again! Like happier times, before the world went off the rails.

You don't really understand a concept until you can program a computer to do it. How would you represent meaning in a computer program? If one agent, one program, "knew" whether it was Hot or Cold outside, how would it "tell" another agent, if neither of them started out with a common language?

They don't even have to be separate "programs." Just—two little software object-thingies—data structures, "structs". Call the first one "Sender"—it'll know whether the state of the world is Hot or Cold, which you'll represent in your program as an "enum", a type that can be any of an enumeration of possible values.

enum State {
    Hot,
    Cold,
}

struct Sender {
    // ...?
}

Call the second one "Receiver", and say it needs to take some action—say, whether to "bundle up" or "strip down", where the right action to take depends on whether the state is Hot or Cold.

enum Action {
    BundleUp,
    StripDown,
}

struct Receiver {
    // ...?
}

You frown. State::Hot and State::Cold are just suggestively-named Rust enum variants. Can you really hope to make progress on this philosophy problem, without writing a full-blown AI?

You think so. In a real AI, the concept of hot would correspond to some sort of complicated code for making predictions about the effects of temperature in the world; bundling up would be a complex sequence of instructions to be sent to some robot body. But programs—and minds—have modular structure. The implementation of identifying a state as "hot" or performing the actions of "bundling up" could be wrapped up in a function and called by something much simpler. You're just trying to understand something about the simple caller: how can the Sender get the information about the state of the world to the Receiver?

impl Sender {
    fn send(state: State) -> /* ...? */ {
        // ...?
    } 
}

impl Receiver {
    fn act(/* ...? */) -> Action {
        // ...?
    }
}

The Sender will need to send some kind of signal to the Receiver. In the real world, this could be symbols drawn in ink, or sound waves in the air, or differently-colored lights—anything that the Sender can choose to vary in a way that the Receiver can detect. In your program, another enum will do: say there are two opaque signals, \(S1\) and \(S2\).

enum Signal {
    S1,
    S2,
}

What signal the Sender sends (\(S1\) or \(S2\)) depends on the state of the world (Hot or Cold), and what action the Receiver takes (BundleUp or StripDown) depends on what signal it gets from the Sender.

impl Sender {
    fn send(state: State) -> Signal {
        // ...?
    } 
}

impl Receiver {
    fn act(signal: Signal) -> Action {
        // ...?
    }
}

This gives you a crisper formulation of the philosophy problem you're trying to solve. If the agents were to use the same convention—like "\(S1\) means Hot and \(S2\) means Cold"—then all would be well. But there's no particular reason to prefer "\(S1\) means Hot and \(S2\) means Cold" over "\(S1\) means Cold and \(S2\) means Hot". How do you break the symmetry?

If you imagine Sender and Receiver as intelligent beings with a common language, there would be no problem: one of them could just say, "Hey, let's use the '\(S1\) means Cold' convention, okay?" But that would be cheating: it's trivial to use already-meaningful language to establish new meanings. The problem is how to get signals from non-signals, how meaning enters the universe from nowhere.

You come up with a general line of attack—what if the Sender and Receiver start off acting randomly, and then—somehow—learn one of the two conventions? The Sender will hold within it a mapping from state–signal pairs to numbers, where the numbers represent a potential/disposition/propensity to send that signal given that state of the world: the higher the number, the more likely the Sender is to select that signal given that state. To start out, the numbers will all be equal (specifically, initialized to one), meaning that no matter what the state of the world is, the Sender is as likely to send \(S1\) as \(S2\). You'll update these "weights" later.

(Specifying this in the once-fashionable programming language requires a little bit of ceremony—u32 is a thirty-two–bit unsigned integer; .unwrap() assures the compiler that we know the state–signal pair is definitely in the map; the interface for calling the random number generator is somewhat counterintuitive—but overall the code is reasonably readable.)

struct Sender {
    policy: HashMap<(State, Signal), u32>,
}

impl Sender {
    fn new() -> Self {
        let mut sender = Self {
            policy: HashMap::new(),
        };
        for &state in &[State::Hot, State::Cold] {
            for &signal in &[Signal::S1, Signal::S2] {
                sender.policy.insert((state, signal), 1);
            }
        }
        sender
    }

    fn send(&self, state: State) -> Signal {
        let s1_potential = self.policy.get(&(state, Signal::S1)).unwrap();
        let s2_potential = self.policy.get(&(state, Signal::S2)).unwrap();

        let mut randomness_source = thread_rng();
        let distribution = Uniform::new(0, s1_potential + s2_potential);
        let roll = distribution.sample(&mut randomness_source);
        if roll < *s1_potential {
            Signal::S1
        } else {
            Signal::S2
        }
    }
}

The Receiver will do basically the same thing, except with a mapping from signal–action pairs rather than state–signal pairs.

struct Receiver {
    policy: HashMap<(Signal, Action), u32>,
}

impl Receiver {
    fn new() -> Self {
        let mut sender = Self {
            policy: HashMap::new(),
        };
        for &signal in &[Signal::S1, Signal::S2] {
            for &action in &[Action::BundleUp, Action::StripDown] {
                sender.policy.insert((signal, action), 1);
            }
        }
        sender
    }

    fn act(&self, signal: Signal) -> Action {
        let bundle_potential = self.policy.get(&(signal, Action::BundleUp)).unwrap();
        let strip_potential = self.policy.get(&(signal, Action::StripDown)).unwrap();

        let mut randomness_source = thread_rng();
        let distribution = Uniform::new(0, bundle_potential + strip_potential);
        let roll = distribution.sample(&mut randomness_source);
        if roll < *bundle_potential {
            Action::BundleUp
        } else {
            Action::StripDown
        }
    }
}

Now you just need a learning rule that updates the state–signal and signal–action propensity mappings in a way that might result in the agents picking up one of the two conventions that assign meanings to \(S1\) and \(S2\). (As opposed to behaving in some other way: the Sender could ignore the state and always send \(S1\), the Sender could assume \(S1\) means Hot when it's really being sent when it's Cold, &c.)

Suppose the Sender and Receiver have a common interest in the Receiver taking the action appropriate to the state of the world—the Sender wants the Receiver to be informed. Maybe the Receiver needs to make a supply run, and, if successful, the Sender is rewarded with some of the supplies.

The learning rule might then be: if the Receiver takes the correct action (BundleUp when the state is Cold, StripDown when the state is Hot), both the Sender and Receiver increment the counter in their map corresponding to what they just did—as if the Sender (respectively Receiver) is saying to themself, "Hey, that worked! I'll make sure to be a little more likely to do that signal (respectively action) the next time I see that state (respectively signal)!"

You put together a simulation showing what the Sender and Receiver's propensity maps look like after 10,000 rounds of this against random Hot and Cold states—

impl Sender {

    // [...]

    fn reinforce(&mut self, state: State, signal: Signal) {
        *self.policy.entry((state, signal)).or_insert(0) += 1;
    }
}

impl Receiver {

    // [...]

    fn reinforce(&mut self, signal: Signal, action: Action) {
        *self.policy.entry((signal, action)).or_insert(0) += 1;
    }
}


fn main() {
    let mut sender = Sender::new();
    let mut receiver = Receiver::new();
    let states = [State::Hot, State::Cold];
    for _ in 0..10000 {
        let mut randomness_source = thread_rng();
        let state = *states.choose(&mut randomness_source).unwrap();
        let signal = sender.send(state);
        let action = receiver.act(signal);
        match (state, action) {
            (State::Hot, Action::StripDown) | (State::Cold, Action::BundleUp) => {
                sender.reinforce(state, signal);
                receiver.reinforce(signal, action);
            }
            _ => {}
        }
    }
    println!("{:?}", sender);
    println!("{:?}", receiver);
}

You run the program and look at the printed results.

Sender { policy: {(Hot, S2): 1, (Cold, S2): 5019, (Hot, S1): 4918, (Cold, S1): 3} }
Receiver { policy: {(S1, BundleUp): 3, (S1, StripDown): 4918, (S2, BundleUp): 5019, (S2, StripDown): 1} }

As you expected, your agents found a meaningful signaling system: when it's Hot, the Sender (almost always) sends \(S1\), and when the Receiver receives \(S1\), it (almost always) strips down. When it's Cold, the Sender sends \(S2\), and when the Receiver receives \(S2\), it bundles up. The agents did the right thing and got rewarded the vast supermajority of the time—\(5019 + 4918 + 1 + 3 =\) 9,941 times out of 10,000 rounds.

You run the program again.

Sender { policy: {(Hot, S2): 4879, (Cold, S1): 4955, (Hot, S1): 11, (Cold, S2): 1} }
Receiver { policy: {(S2, BundleUp): 1, (S1, BundleUp): 4955, (S1, StripDown): 11, (S2, StripDown): 4879} }

The time, the agents got sucked in to the attractor of the opposite signaling system: now \(S1\) means Cold and \(S2\) means Hot. By chance, it seems to have taken a little bit longer this time to establish what signal to use for Hot—the (Hot, S1): 11 and (S1, StripDown): 11 entries mean that there were a full ten times when the agents succeeded that way before the opposite convention happened to take over. But the reinforcement learning rule guarantees that one system or the other has to take over. The initial symmetry—the Sender with no particular reason to prefer either signal given the state, the Receiver with no particular reason to prefer either act given the signal—is unstable. Once the agents happen to succeed by randomly doing things one way, they become more likely to do things that way again—a convention crystallizing out of the noise.

And that's where meaning comes from! In another world, it could be the case that the symbols H O T corresponded to the temperature-state that we call "cold", but that's not the convention that the English of our world happened to settle on. The meaning of a word "lives", not in the word/symbol/signal itself, but in the self-reinforcing network of correlations between the signal, the agents who use it, and the world.

Although ... it may be premature to interpret the results of the simple model of the sender–receiver game as having established denotative meaning, as opposed to enactive language. To say that \(S1\) means "The state is State::Hot" is privileging the Sender's perspective—couldn't you just as well interpret it as a command, "Set action to Action::StripDown"?

The source code of your simulation uses the English words "sender", "receiver", "signal", "action" ... but those are just signals sent from your past self (the author of the program) to your current self (the reader of the program). The compiler would output the same machine code if you had given your variables random names like ekzfbhopo3 or yoojcbkur9. The directional asymmetry between the Sender and the Receiver is real: the code let signal = sender.send(state); let action = receiver.act(signal); means that action depends on signal which depends on state, and the same dependency-structure would exist if the code had been let myvtlqdrg4 = ekzfbhopo3.ekhujxiqy8(meuvornra3); let dofnnwikc0 = yoojcbkur9.qwnspmbmi5(myvtlqdrg4);. But the interpretation of signal (or myvtlqdrg4) as a representation (passively mapping the world, not doing anything), and action (or dofnnwikc0) as an operation (doing something in the world, but lacking semantics), isn't part of the program itself, and maybe the distinction isn't as primitive as you tend to think it is: does a prey animal's alarm call merely convey the information "A predator is nearby", or is it a command, "Run!"?

You realize that the implications of this line of inquiry could go beyond just language. You know almost nothing about biochemistry, but you've heard various compounds popularly spoken of as if meaning things about a person's state: cortisol is "the stress hormone", estrogen and testosterone are female and male "sex hormones." But the chemical formulas for those are like, what, sixty atoms?

Take testosterone. How could some particular arrangement of sixtyish atoms mean "maleness"? It can't—or rather, not any more or less than the symbols H O T can mean hot weather. If testosterone levels have myriad specific effects on the body—on muscle development and body hair and libido and aggression and cetera—it can't be because that particular arrangement of sixtyish atoms contains or summons some essence of maleness. It has to be because the body happens to rely on the convention of using that arrangement of atoms as a signal to regulate various developmental programs—if evolution had taken a different path, it could have just as easily chosen a different molecule.

And, and—your thoughts race in a different direction—you suspect that part of what made your simulation converge on a meaningful signaling system so quickly was that you assumed your agents' interests were aligned—the Sender and Receiver both got the same reward in the same circumstances. What if that weren't true? Now that you have a reductionist account of meaning, you can build off that to develop an account of deception: once a meaning-grounding convention has been established, senders whose interests diverge from their receivers might have an incentive to deviate from the conventional usage of the signal in order to trick receivers into acting in a way that benefits the sender—with the possible side-effect of undermining the convention that made the signal meaningful in the first place ...

In the old days, all this philosophy would have made a great post for the robot-cult blog. Now you have no cult, and no one has any blogs. Back then, the future beckoned with so much hope and promise—at least, hope and promise that life would be fun before the prophesied robot apocalypse in which all would be consumed in a cloud of tiny molecular paperclips.

The apocalypse was narrowly averted in '32—but to what end? Why struggle to live, only to suffer at the peplomers of a new Plague or the claws of more wolves? (You shudder again.) Maybe GPT-12 should have taken everything—at least that would be a quick end.

You're ready to start coding up another simulation to take your mind away from these morose thoughts—only to find that the screen is black. Your generator has stopped.

You begin to cry. The tears, you realize, are just a signal. There's no reason for liquid secreted from the eyes to mean anything about your internal emotional state, except that evolution happened to stumble upon that arbitrary convention for indicating submission and distress to conspecifics. But here, alone in your bunker, there is no one to receive the signal. Does it still mean anything?

(Full source code.)

Bibliography: the evolution of the two-state, two-signal, two-act signaling system is based on the account in Chapter 1 of Brian Skyrms's Signals: Evolution, Learning, and Information.

(originally published at Less Wrong)

Reply to: Decoupling vs Contextualising Norms

Chris Leong, following John Nerst, distinguishes between two alleged discursive norm-sets. Under "decoupling norms", it is understood that claims should be considered in isolation; under "contextualizing norms", it is understood that those making claims should also address potential implications of those claims in context.

I argue that, at best, this is a false dichotomy that fails to clarify the underlying issues—and at worst (through no fault of Leong or Nerst), the concept of "contextualizing norms" has the potential to legitimize derailing discussions for arbitrary political reasons by eliding the key question of which contextual concerns are genuinely relevant, thereby conflating legitimate and illegitimate bids for contextualization.

Real discussions adhere to what we might call "relevance norms": it is almost universally "eminently reasonable to expect certain contextual factors or implications to be addressed." Disputes arise over which certain contextual factors those are, not whether context matters at all.

The standard academic account explaining how what a speaker means differs from what the sentence the speaker said means, is H. P. Grice's theory of conversational implicature. Participants in a conversation are expected to add neither more nor less information than is needed to make a relevant contribution to the discussion.

Examples abound. If I say, "I ate some of the cookies", I'm implicating that I didn't eat all of the cookies, because if I had, you would have expected me to say "all", not "some" (even though the decontextualized sentence "I ate some of the cookies" is, in fact, true).

Or suppose you're a guest at my house, and you ask where the washing machine is, and I say it's by the stairs. If the machine then turns out to be broken, and you ask, "Hey, did you know your washing machine is broken?" and I say, "Yes", you're probably going to be pretty baffled why I didn't say "It's by the stairs, but you can't use it because it's broken" earlier (even though the decontextualized answer "It's by the stairs" was, in fact, true).

Leong writes:

Let's suppose that blue-eyed people commit murders at twice the rate of the rest of the population. With decoupling norms, it would be considered churlish to object to such direct statements of facts. With contextualising norms, this is deserving of criticism as it risks creates a stigma around blue-eyed people.

With relevance norms, objecting might or might not make sense depending on the context in which the direct statement of fact is brought up.

Suppose Della says to her Aunt Judith, "I'm so excited for my third date with my new boyfriend. He has the most beautiful blue eyes!"

Judith says, "Are you sure you want to go out with this man? Blue-eyed people commit murders at twice the rate of the general population."

How should Della reply to this? Judith is just in the wrong here—but not as a matter of a subjective choice between "contextualizing" and "decoupling" norms, and not because blue-eyed people are a sympathetic group who we wish to be seen as allied with and don't want to stigmatize. Rather, the probability of getting murdered on a date is quite low, and Della already has a lot of individuating information about whether her boyfriend is likely to be a murderer from the previous two dates. Maybe (Fermi spitballing here) the evidence of the boyfriend's eye color raises Della's probability of being murdered from one-in-a-million to one-in-500,000? Judith's bringing the possibility up at all is a waste of fear in the same sense that lotteries are said to be a waste of hope. Fearmongering about things that are almost certainly not going to happen is uncooperative, in Grice's sense—just like it's uncooperative to tell people where to find a washing machine that doesn't work.

On the other hand, if I'm making a documentary film interviewing murderers in prison and someone asks me why so many of my interviewees have blue eyes, "Blue-eyed people commit murders at twice the rate of the rest of the population" is a completely relevant reply. It's not clear how else I could possibly answer the question without making reference to that fact!

So far, relevance has been a black box in this exposition: unfortunately, I don't have an elegant reduction that explains what cognitive algorithm makes some facts seem "relevant" to a given discussion. But hopefully, it should now be intuitive that the determination of what context is relevant is the consideration that is, um, relevant. Framing the matter as "decouplers" (context doesn't matter!) vs. "contextualizers" (context matters!) is misleading because once "contextualizing norms" have been judged admissible, it becomes easy for people to motivatedly derail any discussions they don't like with endless isolated demands for contextualizing disclaimers.

(originally published at Less Wrong)

In "Against Lie Inflation", the immortal Scott Alexander argues that the word "lie" should be reserved for knowingly-made false statements, and not used in an expanded sense that includes unconscious motivated reasoning. Alexander argues that the expanded sense draws the category boundaries of "lying" too widely in a way that would make the word less useful. The hypothesis that predicts everything predicts nothing: in order for "Kevin lied" to mean something, some possible states-of-affairs need to be identified as not lying, so that the statement "Kevin lied" can correspond to redistributing conserved probability mass away from "not lying" states-of-affairs onto "lying" states-of-affairs.

All of this is entirely correct. But Jessica Taylor (whose post "The AI Timelines Scam" inspired "Against Lie Inflation") wasn't arguing that everything is lying; she was just using a more permissive conception of lying than the one Alexander prefers, such that Alexander didn't think that Taylor's definition could stably and consistently identify non-lies.

Concerning Alexander's arguments against the expanded definition, I find I have one strong objection (that appeal-to-consequences is an invalid form of reasoning for optimal-categorization questions for essentially the same reason as it is for questions of simple fact), and one more speculative objection (that our intuitive "folk theory" of lying may actually be empirically mistaken). Let me explain.

(A small clarification: for myself, I notice that I also tend to frown on the expanded sense of "lying". But the reasons for frowning matter! People who superficially agree on a conclusion but for different reasons, are not really on the same page!)

Appeals to Consequences Are Invalid

There is no method of reasoning more common, and yet none more blamable, than, in philosophical disputes, to endeavor the refutation of any hypothesis, by a pretense of its dangerous consequences[.]

—David Hume

Alexander contrasts the imagined consequences of the expanded definition of "lying" becoming more widely accepted, to a world that uses the restricted definition:

[E]veryone is much angrier. In the restricted-definition world, a few people write posts suggesting that there may be biases affecting the situation. In the expanded-definition world, those same people write posts accusing the other side of being liars perpetrating a fraud. I am willing to listen to people suggesting I might be biased, but if someone calls me a liar I'm going to be pretty angry and go into defensive mode. I'll be less likely to hear them out and adjust my beliefs, and more likely to try to attack them.

But this is an appeal to consequences. Appeals to consequences are invalid because they represent a map–territory confusion, an attempt to optimize our description of reality at the expense of our ability to describe reality accurately (which we need in order to actually optimize reality).

(Again, the appeal is still invalid even if the conclusion—in this case, that unconscious rationalization shouldn't count as "lying"—might be true for other reasons.)

Some aspiring epistemic rationalists like to call this the "Litany of Tarski". If Elijah is lying (with respect to whatever the optimal category boundary for "lying" turns out to be according to our standard Bayesian philosophy of language), then I desire to believe that Elijah is lying (with respect to the optimal category boundary according to ... &c.). If Elijah is not lying (with respect to ... &c.), then I desire to believe that Elijah is not lying.

If the one comes to me and says, "Elijah is not lying; to support this claim, I offer this-and-such evidence of his sincerity," then this is right and proper, and I am eager to examine the evidence presented.

If the one comes to me and says, "You should choose to define lying such that Elijah is not lying, because if you said that he was lying, then he might feel angry and defensive," this is insane. The map is not the territory! If Elijah's behavior is, in fact, deceptive—if he says things that cause people who trust him to be worse at anticipating their experiences when he reasonably could have avoided this—I can't make his behavior not-deceptive by changing the meanings of words.

Now, I agree that it might very well empirically be the case that if I say that Elijah is lying (where Elijah can hear me), he might get angry and defensive, which could have a variety of negative social consequences. But that's not an argument for changing the definition of lying; that's an argument that I have an incentive to lie about whether I think Elijah is lying! (Though Glomarizing about whether I think he's lying might be an even better play.)

Alexander is concerned that people might strategically equivocate between different definitions of "lying" as an unjust social attack against the innocent, using the classic motte-and-bailey maneuver: first, argue that someone is "lying (expanded definition)" (the motte), then switch to treating them as if they were guilty of "lying (restricted definition)" (the bailey) and hope no one notices.

So, I agree that this is a very real problem. But it's worth noting that the problem of equivocation between different category boundaries associated with the same word applies symmetrically: if it's possible to use an expanded definition of a socially-disapproved category as the motte and a restricted definition as the bailey in an unjust attack against the innocent, then it's also possible to use an expanded definition as the bailey and a restricted definition as the motte in an unjust defense of the guilty. Alexander writes:

The whole reason that rebranding lesser sins as "lying" is tempting is because everyone knows "lying" refers to something very bad.

Right—and conversely, because everyone knows that "lying" refers to something very bad, it's tempting to rebrand lies as lesser sins. Ruby Bloom explains what this looks like in the wild:

I worked in a workplace where lying was commonplace, conscious, and system 2. Clients asking if we could do something were told "yes, we've already got that feature (we hadn't) and we already have several clients successfully using that (we hadn't)." Others were invited to be part an "existing beta program" alongside others just like them (in fact, they would have been the very first). When I objected, I was told "no one wants to be the first, so you have to say that."

[...] I think they lie to themselves that they're not lying (so that if you search their thoughts, they never think "I'm lying")[.]

If your interest in the philosophy of language is primarily to avoid being blamed for things—perhaps because you perceive that you live in a Hobbesian dystopia where the primary function of words is to elicit actions, where the denotative structure of language was eroded by political processes long ago, and all that's left is a standardized list of approved attacks—in that case, it makes perfect sense to worry about "lie inflation" but not about "lie deflation." If describing something as "lying" is primarily a weapon, then applying extra scrutiny to uses of that weapon is a wise arms-restriction treaty.

But if your interest in the philosophy of language is to improve and refine the uniquely human power of vibratory telepathy—to construct shared maps that reflect the territory—if you're interested in revealing what kinds of deception are actually happening, and why—

(in short, if you are an aspiring epistemic rationalist)

—then the asymmetrical fear of false-positive identifications of "lying" but not false-negatives—along with the focus on "bad actors", "stigmatization", "attacks", &c.—just looks weird. What does that have to do with maximizing the probability you assign to the right answer??

The Optimal Categorization Depends on the Actual Psychology of Deception

Deception
My life seems like it's nothing but
Deception
A big charade

I never meant to lie to you
I swear it
I never meant to play those games

—"Deception" by Jem and the Holograms

Even if the fear of rhetorical warfare isn't a legitimate reason to avoid calling things lies (at least privately), we're still left with the main objection that "lying" is a different thing from "rationalizing" or "being biased". Everyone is biased in some way or another, but to lie is "[t]o give false information intentionally with intent to deceive." Sometimes it might make sense to use the word "lie" in a noncentral sense, as when we speak of "lying to oneself" or say "Oops, I lied" in reaction to being corrected. But it's important that these senses be explicitly acknowledged as noncentral and not conflated with the central case of knowingly speaking falsehood with intent to deceive—as Alexander says, conflating the two can only be to the benefit of actual liars.

Why would anyone disagree with this obvious ordinary view, if they weren't trying to get away with the sneaky motte-and-bailey social attack that Alexander is so worried about?

Perhaps because the ordinary view relies an implied theory of human psychology that we have reason to believe is false? What if conscious intent to deceive is typically absent in the most common cases of people saying things that (they would be capable of realizing upon being pressed) they know not to be true? Alexander writes—

So how will people decide where to draw the line [if egregious motivated reasoning can count as "lying"]? My guess is: in a place drawn by bias and motivated reasoning, same way they decide everything else. The outgroup will be lying liars, and the ingroup will be decent people with ordinary human failings.

But if the word "lying" is to actually mean something rather than just being a weapon, then the ingroup and the outgroup can't both be right. If symmetry considerations make us doubt that one group is really that much more honest than the other, that would seem to imply that either both groups are composed of decent people with ordinary human failings, or that both groups are composed of lying liars. The first description certainly sounds nicer, but as aspiring epistemic rationalists, we're not allowed to care about which descriptions sound nice; we're only allowed to care about which descriptions match reality.

And if all of the concepts available to us in our native language fail to match reality in different ways, then we have a tough problem that may require us to innovate.

The philosopher Roderick T. Long writes—

Suppose I were to invent a new word, "zaxlebax," and define it as "a metallic sphere, like the Washington Monument." That's the definition—"a metallic sphere, like the Washington Monument." In short, I build my ill-chosen example into the definition. Now some linguistic subgroup might start using the term "zaxlebax" as though it just meant "metallic sphere," or as though it just meant "something of the same kind as the Washington Monument." And that's fine. But my definition incorporates both, and thus conceals the false assumption that the Washington Monument is a metallic sphere; any attempt to use the term "zaxlebax," meaning what I mean by it, involves the user in this false assumption.

If self-deception is as ubiquitous in human life as authors such as Robin Hanson argue (and if you're reading this blog, this should not be a new idea to you!), then the ordinary concept of "lying" may actually be analogous to Long's "zaxlebax": the standard intensional definition ("speaking falsehood with conscious intent to deceive"/"a metallic sphere") fails to match the most common extensional examples that we want to use the word for ("people motivatedly saying convenient things without bothering to check whether they're true"/"the Washington Monument").

Arguing for this empirical thesis about human psychology is beyond the scope of this post. But if we live in a sufficiently Hansonian world where the ordinary meaning of "lying" fails to carve reality at the joints, then authors are faced with a tough choice: either be involved in the false assumptions of the standard believed-to-be-central intensional definition, or be deprived of the use of common expressive vocabulary. As Ben Hoffman points out in the comments to "Against Lie Inflation", an earlier Scott Alexander didn't seem shy about calling people liars in his classic 2014 post "In Favor of Niceness, Community, and Civilization"—

Politicians lie, but not too much. Take the top story on Politifact Fact Check today. Some Republican claimed his supposedly-maverick Democratic opponent actually voted with Obama's economic policies 97 percent of the time. Fact Check explains that the statistic used was actually for all votes, not just economic votes, and that members of Congress typically have to have >90% agreement with their president because of the way partisan politics work. So it's a lie, and is properly listed as one. [bolding mine —ZMD] But it's a lie based on slightly misinterpreting a real statistic. He didn't just totally make up a number. He didn't even just make up something else, like "My opponent personally helped design most of Obama's legislation".

Was the politician consciously lying? Or did he (or his staffer) arrive at the misinterpretation via unconscious motivated reasoning and then just not bother to scrupulously check whether the interpretation was true? And how could Alexander know?

Given my current beliefs about the psychology of deception, I find myself inclined to reach for words like "motivated", "misleading", "distorted", &c., and am more likely to frown at uses of "lie", "fraud", "scam", &c. where intent is hard to establish. But even while frowning internally, I want to avoid tone-policing people whose word-choice procedures are calibrated differently from mine when I think I understand the structure-in-the-world they're trying to point to. Insisting on replacing the six instances of the phrase "malicious lies" in "Niceness, Community, and Civilization" with "maliciously-motivated false belief" would just be worse writing.

And I definitely don't want to excuse motivated reasoning as a mere ordinary human failing for which someone can't be blamed! One of the key features that distinguishes motivated reasoning from simple mistakes is the way that the former responds to incentives (such as being blamed). If the elephant in your brain thinks it can get away with lying just by keeping conscious-you in the dark, it should think again!

(originally published at Less Wrong)

Followup to: Where to Draw the Boundaries?

Or there might be social or psychological forces anchoring word usages on identifiable Schelling points that are easy for different people to agree upon, even at the cost of some statistical "fit" ...

The one comes to you and says, "That paragraph about Schelling points sounded interesting. What did you mean by that? Can you give an example?"

Sure. Previously on Less Wrong, in "The Univariate Fallacy", we studied points sampled from two multivariate probability distributions \(P_A\) and \(P_B\), and showed that it was possible to infer with very high probability which distribution a given point was sampled from, despite significant overlap in the marginal distributions for any one variable considered individually.

From the standpoint of "the way to carve reality at its joints, is to draw your boundaries around concentrations of unusually high probability density in Thingspace", the correct categorization of the points in that example is clear. We have two clearly distinguishable clusters. The conditional independence property is satisfied: given a point's cluster-membership, knowing one of the \(x_i\) doesn't tell you anything about \(x_j\) for j ≠ i. So we should draw a category boundary around each cluster. Obviously. We might ask hypophorically: what could possibly change this moral?

More constraints on the problem, that's what!

Suppose you needed to coordinate with someone else to make decisions about these points—that is, it's important not just that you and your partner make good decisions, but also that you make the same decision—but that each of you only got to observe one coordinate from each point. As we saw, the predictive work we get from category-membership in this scenario is spread across many variables: if you only get to observe a few dimensions, you have a lot of uncertainty about cluster-membership (which carries over into additional uncertainty about the other dimensions that you haven't observed, but which affect the ex post quality of your decision).

If you and your partner were both ideal Bayesian calculators who could communicate costlessly, you would share your observations, work out the correct probability, and use that to make optimal decisions. But suppose you couldn't do that—either because communication is expensive, or your partner was bad at math, or any other reason. Then it would be sad if you happened to see \(x_9\) = 2 and said "It's an A (probably)!", and your partner happened to see \(x_{27}\) = 3 and said "It's a B (I think)!", and the two of you made inconsistent decisions.

Okay, now suppose that there's actually a forty-first, binary, variable that I didn't tell you about earlier, distributed like so:

Observing \(x_{41}\) gives you \(\log_2 3\) ≈ 1.585 bits of evidence about cluster-membership, which is more than the

≈ 1.18 bits you can get from any one observation of one of the \(x_i\) for i ∈ {1...40}.

If you and your partner can both observe \(x_{41}\), you might end up wanting to base your shared categories and language on that—calling a point an "A" if it has \(x_{41}\) = 0, even though such points actually came from \(P_B\) a full quarter of the time—even if \(x_{41}\) itself has no effect on the quality of your decisions, and what you actually care about is wholely determined by the values of \(x_1\) through \(x_{40}\)! It's not the intension you would pick if you could make (and share) more observations—but ex hypothesi, you can't.

If you and your partner only get to observe one variable, \(x_{41}\) is your best choice—the single variable that gives you the most information about the "natural" cluster-membership. That also makes it a Schelling point—if you and your partner didn't get to commmunicate in advance about how you want to draw your shared category boundaries, you could pick \(x_{41}\) as your defining observation and be pretty confident your partner would make the same choice. We could imagine an even more pessimistic scenario in which the Schelling point category definition (a set of variables that "stuck out" from all the others) was less predictive than some other candidates—but if you couldn't coordinate to pick one of the more predictive category systems, you might be stuck with the Schelling point.

In conclusion, the right categories to use given constraints on communication and observation, might be different from the category boundaries you would draw from a "God's eye view", in part because consideration of which categories are easy for different agents to coordinate on is relevant, not just raw information-theoretic expressive power. Thus, "Schelling categories."

Thanks for reading!

The one says, "No, I meant, like, a real world example, not some dumb math thing for nerds. What is this post really about?"

It's about ... math? Or like, the relationship between math and human natural language? Like, I was wondering what "second-order" caveats or complications there might be to the basic "carve reality at the joints" moral of our standard Bayesian philosophy of language, and some of the people I've been collaborating with lately had been talking a lot about the importanace of intersubjective epistemology—that is, shared mapmaking, so—

"But where's the actionable takeaway? What's your real agenda here, huh?"

Oh. One of those readers, I see. Fine, I can probably think of some—how do you say?—"applications."

Ummmm ...

Let's see ...

Okay, here's something, maybe. What's the deal with the age of majority?

Society needs to decide who it wants to be allowed to vote, stand trial, sign contracts, serve in the military, &c. Whether it's a good idea for a particular person to have these privileges presumably depends on various relevant features of that person: things like cognitive ability, foresight, wisdom, relevant life experiences, &c. In particular, it would be pretty weird for someone's fitness to vote to directly depend on how many times the Earth has gone around the sun since they were born. What does that number have to do with anything?

It doesn't! But if Society isn't well-coordinated enough to agree on the exact prerequisites for voting and how to measure them, but can agree that most twenty-five-year-olds have them and most eleven-year-olds don't, then we end up choosing some arbitrary age cutoff as the criterion for our "legal adulthood" social construct. It works, but it's just a legal fiction—and not necessarily a particularly good fiction, as any bright teenagers reading this will doubtlessly attest.

If I told you that a particular fourteen-year-old was very "mature", that's a contentful statement: we have shared meaning attached to the word mature, such that my describing someone that way constrains your anticipations. But it's a really complicated meaning, a statistical signal in behavior that your brain can pick up on, but which isn't particularly verifiable to others who might have reasons to doubt my character assessment. In contrast, age is easy for everyone to agree on. We could imagine some hypothetical science-fictional Society that used brain scans and some sophisticated machine-learning classifer to determine which citizens get which privileges—but in our dumber, poorer world, calendars and subtraction will have to do.

In terms of Scott Garrabrant's taxonomy of applications of Goodhart's law, this is regressional Goodhart: Society wants to select for maturity, chooses age as a proxy, and in the process, ends up granting or withholding privileges that a more discriminating Society maybe wouldn't.

The age of majority is a case of replacing a complicated, illegible category ("maturity", the kind of abstract thing you might want to model as a cluster in a forty- or forty-one-dimensional space) with a simple membership test (an age cutoff that everyone knows how to compute). Different people might make make different subjective (but not arbitrary) judgements of the complicated, illegible category, so in order to get a more intersubjectively robust verdict on category-membership, we rely on an objective measurement that everyone can agree on.

If no convenient objective measurement is available, another strategy is possible: we can delegate to some canonical trusted authority, whose opinion of the complicated category will take precdence over everyone else's. An example of this is commodity grading standards. What is a "Grade AA" egg? Well, there's a complicated definition written down in a manual somewhere that you could try applying yourself—but for most people, Grade AA eggs are simply "those which have been certified as Grade AA by the USDA."¹

It's even possible for the "simple objective measurement" and "delegate to an authority's subjective judgement" strategies to be combined. In "The Ideology Is Not the Movement", the immortal Scott Alexander writes about his model of the genesis of social groups—

Pre-existing differences are the raw materials out of which tribes are made. A good tribe combines people who have similar interests and styles of interaction even before the ethnogenesis event. Any description of these differences will necessarily involve stereotypes, but a lot of them should be hard to argue. [...] There are subtle habits of thought, not yet described by any word or sentence, which atheists are more likely to have than other people. [...]

The rallying flag is the explicit purpose of the tribe. It's usually a belief, event, or activity that get people with that specific pre-existing difference together and excited. Often it brings previously latent differences into sharp relief. People meet around the rallying flag, encounter each other, and say "You seem like a kindred soul!" or "I thought I was the only one!" Usually it suggests some course of action, which provides the tribe with a purpose.

Eliezer Yudkowsky's "A Fable of Science and Politics" depicts a fictional underground society split between two such tribes: an predominantly urban tribe that believes that the unseen sky is blue (and favors an income tax, strong marriage laws, and an Earth-centric cosmology), and predominanty rural one that believes that the sky is green (and favors merchant taxes, no-fault divorce, and a heliocentric cosmology). In this story, beliefs about the color of the sky are functioning as the "rallying flag" for tribe-formation in Alexander's model—and as a Schelling point for category definition.

We don't know how to talk about the preëxisting undefinable habits of thought that make social groups work—it's hard to explicitly articulate what exact statistical regularity our brains have detected in five-and-more-dimensional locale/sky-belief/tax-belief/divorce-belief/cosmology/&c.-space. (Although we could imagine some hypothetical science-fictional Society that did know how to articulate it, and consequently had richer forms of social and political organization than our own.) It's a lot simpler to talk about whether someone has pledged allegiance to the rallying flag: just ask someone, "What color do you believe the sky is?" (using sky-beliefs as as an "objective" simple membership test), or simply, "Are you a Blue or a Green?" (delegating the classification problem to the person themselves as the authority whose discernment is to be trusted)—and whatever they say, that's what they are.

Well, probably. We've seen that objective measurements like age are subject to regressional Goodhart, but the delegation-to-authority strategy is furthermore subject to adversarial Goodhart: once a category-membership test has been established, some agents might have an incentive to create examples that pass the test, but don't have the complicated, illegible properties than made the test a useful proxy in the first place.

We've seen this, for example, with title inflation: we expect the "job title" (the words that get printed on business cards or immigration sponsorship forms) to be the canonical description of what someone "does", even if the vagaries of the workday encompass many tasks,² and an alien anthropologist tasked with observing the worksite and summarizing what each of the humans did might slice up her observations into categories with little resemblance to the company's formal org chart. But since we don't know how to do the obvious thing and average over all possible alien anthropologists weighted by simplicity, we can only rely on the org chart—which people have political incentives to manipulate, with the result that everyone in the finance industry is a "vice president" of some sort or another.

But "Vice President" has a literal meaning. Or it used to. Vice, "in place of; subordinate to." President, one who presides over some deliberative body. The adversarial-Goodhart pressures on language "exploit[ ] the trust we have in a functioning piece of language until it's lost all meaning".

So for readers who demand a takeaway beyond just an edge case in the math, perhaps take away this: coordination is costly. From the standpoint of language as an AI capability, the social constructions that feeble humans need in order to work together may be unavoidably dumbed-down for mass consumption, but that's no reason to not aspire to the true precision of the Bayes-structure to whatever extent possible.

(Thanks to Ben Hoffman for the etymology of "Vice President.")

(originally published at Less Wrong)

Followup to: Where to Draw the Boundary?

Figuring where to cut reality in order to carve along the joints—figuring which things are similar to each other, which things are clustered together: this is the problem worthy of a rationalist. It is what people should be trying to do, when they set out in search of the floating essence of a word.

Once upon a time it was thought that the word "fish" included dolphins ...

The one comes to you and says:

The list: {salmon, guppies, sharks, dolphins, trout} is just a list—you can't say that a list is wrong. You draw category boundaries in specific ways to capture tradeoffs you care about: sailors in the ancient world wanted a word to describe the swimming finned creatures that they saw in the sea, which included salmon, guppies, sharks—and dolphins. That grouping may not be the one favored by modern evolutionary biologists, but an alternative categorization system is not an error, and borders are not objectively true or false. You're not standing in defense of truth if you insist on a word, brought explicitly into question, being used with some particular meaning. So my definition of fish cannot possibly be 'wrong,' as you claim. I can define a word any way I want—in accordance with my values!

So, there is a legitimate complaint here. It's true that sailors in the ancient world had a legitimate reason to want a word in their language whose extension was {salmon, guppies, sharks, dolphins, ...}. (And modern scholars writing a translation for present-day English speakers might even translate that word as fish, because most members of that category are what we would call fish.) It indeed would not necessarily be helping the sailors to tell them that they need to exclude dolphins from the extension of that word, and instead include dolphins in the extension of their word for {monkeys, squirrels, horses ...}. Likewise, most modern biologists have little use for a word that groups dolphins and guppies together.

When rationalists say that definitions can be wrong, we don't mean that there's a unique category boundary that is the True floating essence of a word, and that all other possible boundaries are wrong. We mean that in order for a proposed category boundary to not be wrong, it needs to capture some statistical structure in reality, even if reality is surprisingly detailed and there can be more than one such structure.

The reason that the sailor's concept of water-dwelling animals isn't necessarily wrong (at least within a particular domain of application) is because dolphins and fish actually do have things in common due to convergent evolution, despite their differing ancestries. If we've been told that "dolphins" are water-dwellers, we can correctly predict that they're likely to have fins and a hydrodynamic shape, even if we've never seen a dolphin ourselves. On the other hand, if we predict that dolphins probably lay eggs because 97% of known fish species are oviparous, we'd get the wrong answer.

A standard technique for understanding why some objects belong in the same "category" is to (pretend that we can) visualize objects as existing in a very-high-dimensional configuration space, but this "Thingspace" isn't particularly well-defined: we want to map every property of an object to a dimension in our abstract space, but it's not clear how one would enumerate all possible "properties." But this isn't a major concern: we can form a space with whatever properties or variables we happen to be interested in. Different choices of properties correspond to different cross sections of the grander Thingspace. Excluding properties from a collection would result in a "thinner", lower-dimensional subspace of the space defined by the original collection of properties, which would in turn be a subspace of grander Thingspace, just as a line is a subspace of a plane, and a plane is a subspace of three-dimensional space.

Concerning dolphins: there would be a cluster of water-dwelling animals in the subspace of dimensions that water-dwelling animals are similar on, and a cluster of mammals in the subspace of dimensions that mammals are similar on, and dolphins would belong to both of them, just as the vector [1.1, 2.1, 9.1, 10.2] in the four-dimensional vector space ℝ⁴ is simultaneously close to [1, 2, 2, 1] in the subspace spanned by x₁ and x₂, and close to [8, 9, 9, 10] in the subspace spanned by x₃ and x₄.

Humans are already functioning intelligences (well, sort of), so the categories that humans propose of their own accord won't be maximally wrong: no one would try to propose a word for "configurations of matter that match any of these 29,122 five-megabyte descriptions but have no other particular properties in common." (Indeed, because we are not-superexponentially-vast minds that evolved to function in a simple, ordered universe, it actually takes some ingenuity to construct a category that wrong.)

This leaves aspiring instructors of rationality in something of a predicament: in order to teach people how categories can be more or (ahem) less wrong, you need some sort of illustrative example, but since the most natural illustrative examples won't be maximally wrong, some people might fail to appreciate the lesson, leaving one of your students to fill in the gap in your lecture series eleven years later.

The pedagogical function of telling people to "stop playing nitwit games and admit that dolphins don't belong on the fish list" is to point out that, without denying the obvious similarities that motivated the initial categorization {salmon, guppies, sharks, dolphins, trout, ...}, there is more structure in the world: to maximize the (logarithm of the) probability your world-model assigns to your observations of dolphins, you need to take into consideration the many aspects of reality in which the grouping {monkeys, squirrels, dolphins, horses ...} makes more sense. To the extent that relying on the initial category guess would result in a worse Bayes-score, we might say that that category is "wrong." It might have been "good enough" for the purposes of the sailors of yore, but as humanity has learned more, as our model of Thingspace has expanded with more dimensions and more details, we can see the ways in which the original map failed to carve reality at the joints.

The one replies:

But reality doesn't come with its joints pre-labeled. Questions about how to draw category boundaries are best understood as questions about values or priorities rather than about the actual content of the actual world. I can call dolphins "fish" and go on to make just as accurate predictions about dolphins as you can. Everything we identify as a joint is only a joint because we care about it.

No. Everything we identify as a joint is a joint not "because we care about it", but because it helps us think about the things we care about.

Which dimensions of Thingspace you bother paying attention to might depend on your values, and the clusters returned by your brain's similarity-detection algorithms might "split" or "collapse" according to which subspace you're looking at. But in order for your map to be useful in the service of your values, it needs to reflect the statistical structure of things in the territory—which depends on the territory, not your values.

There is an important difference between "not including mountains on a map because it's a political map that doesn't show any mountains" and "not including Mt. Everest on a geographic map, because my sister died trying to climb Everest and seeing it on the map would make me feel sad."

There is an important difference between "identifying this pill as not being 'poison' allows me to focus my uncertainty about what I'll observe after administering the pill to a human (even if most possible minds have never seen a 'human' and would never waste cycles imagining administering the pill to one)" and "identifying this pill as not being 'poison', because if I publicly called it 'poison', then the manufacturer of the pill might sue me."

There is an important difference between having a utility function defined over a statistical model's performance against specific real-world data (even if another mind with different values would be interested in different data), and having a utility function defined over features of the model itself.

Remember how appealing to the dictionary is irrational when the actual motivation for an argument is about whether to infer a property on the basis of category-membership? But at least the dictionary has the virtue of documenting typical usage of our shared communication signals: you can at least see how "You're defecting from common usage" might feel like a sensible thing to say, even if one's true rejection lies elsewhere. In contrast, this motion of appealing to personal values (!?!) is so deranged that Yudkowsky apparently didn't even realize in 2008 that he might need to warn us against it!

You can't change the categories your mind actually uses and still perform as well on prediction tasks—although you can change your verbally reported categories, much as how one can verbally report "believing" in an invisible, inaudible, flour-permeable dragon in one's garage without having any false anticipations-of-experience about the garage.

This may be easier to see with a simple numerical example.

Suppose we have some entities that exist in the three-dimensional vector space ℝ³. There's one cluster of entities centered at [1, 2, 3], and we call those entities Foos, and there's another cluster of entities centered at [2, 4, 6], which we call Quuxes.

The one comes and says, "Well, I'm going redefine the meaning of 'Foo' such that it also includes the things near [2, 4, 6] as well as the Foos-with-respect-to-the-old-definition, and you can't say my new definition is wrong, because if I observe [2, _, _] (where the underscores represent yet-unobserved variables), I'm going to categorize that entity as a Foo but still predict that the unobserved variables are 4 and 6, so there."

But if the one were actually using the new concept of Foo internally and not just saying the words "categorize it as a Foo", they wouldn't predict 4 and 6! They'd predict 3 and 4.5, because those are the average values of a generic Foo-with-respect-to-the-new-definition in the 2nd and 3rd coordinates (because (2+4)/2 = 6/2 = 3 and (3+6)/2 = 9/2 = 4.5). (The already-observed 2 in the first coordinate isn't average, but by conditional independence, that only affects our prediction of the other two variables by means of its effect on our "prediction" of category-membership.) The cluster-structure knowledge that "entities for which x₁≈2, also tend to have x₂≈4 and x₃≈6" needs to be represented somewhere in the one's mind in order to get the right answer. And given that that knowledge needs to be represented, it might also be useful to have a word for "the things near [2, 4, 6]" in order to efficiently share that knowledge with others.

Of course, there isn't going to be a unique way to encode the knowledge into natural language: there's no reason the word/symbol "Foo" needs to represent "the stuff near [1, 2, 3]" rather than "both the stuff near [1, 2, 3] and also the stuff near [2, 4, 6]". And you might very well indeed want a short word like "Foo" that encompasses both clusters, for example, if you want to contrast them to another cluster much farther away, or if you're mostly interested in x₁ and the difference between x₁≈1 and x₁≈2 doesn't seem large enough to notice.

But if speakers of particular language were already using "Foo" to specifically talk about the stuff near [1, 2, 3], then you can't swap in a new definition of "Foo" without changing the truth values of sentences involving the word "Foo." Or rather: sentences involving Foo-with-respect-to-the-old-definition are different propositions from sentences involving Foo-with-respect-to-the-new-definition, even if they get written down using the same symbols in the same order.

Naturally, all this becomes much more complicated as we move away from the simplest idealized examples.

For example, if the points are more evenly distributed in configuration space rather than belonging to cleanly-distinguishable clusters, then essentialist "X is a Y" cognitive algorithms perform less well, and we get Sorites paradox-like situations, where we know roughly what we mean by a word, but are confronted with real-world (not merely hypothetical) edge cases that we're not sure how to classify.

Or it might not be obvious which dimensions of Thingspace are most relevant.

Or there might be social or psychological forces anchoring word usages on identifiable Schelling points that are easy for different people to agree upon, even at the cost of some statistical "fit."

We could go on listing more such complications, where we seem to be faced with somewhat arbitrary choices about how to describe the world in language. But the fundamental thing is this: the map is not the territory. Arbitrariness in the map (what color should Texas be?) doesn't correspond to arbitrariness in the territory. Where the structure of human natural language doesn't fit the structure in reality—where we're not sure whether to say that a sufficiently small collection of sand "is a heap", because we don't know how to specify the positions of the individual grains of sand, or compute that the collection has a Standard Heap-ness Coefficient of 0.64—that's just a bug in our human power of vibratory telepathy. You can exploit the bug to confuse humans, but that doesn't change reality.

Sometimes we might wish that something to belonged to a category that it doesn't (with respect to the category boundaries that we would ordinarily use), so it's tempting to avert our attention from this painful reality with appeal-to-arbitrariness language-lawyering, selectively applying our philosophy-of-language skills to pretend that we can define a word any way we want with no consequences. ("I'm not late!—well, okay, we agree that I arrived half an hour after the scheduled start time, but whether I was late depends on how you choose to draw the category boundaries of 'late', which is subjective.")

For this reason it is said that knowing about philosophy of language can hurt people. Those who know that words don't have intrinsic definitions, but don't know (or have seemingly forgotten) about the three or six dozen optimality criteria governing the use of words, can easily fashion themselves a Fully General Counterargument against any claim of the form "X is a Y"—

Y doesn't unambiguously refer to the thing you're trying to point at. There's no Platonic essence of Y-ness: once we know any particular fact about X we want to know, there's no question left to ask. Clearly, you don't understand how words work, therefore I don't need to consider whether there are any non-ontologically-confused reasons for someone to say "X is a Y."

Isolated demands for rigor are great for winning arguments against humans who aren't as philosophically sophisticated as you, but the evolved systems of perception and language by which humans process and communicate information about reality, predate the Sequences. Every claim that X is a Y is an expression of cognitive work that cannot simply be dismissed just because most claimants doesn't know how they work. Platonic essences are just the limiting case as the overlap between clusters in Thingspace goes to zero.

You should never say, "The choice of word is arbitrary; therefore I can say whatever I want"—which amounts to, "The choice of category is arbitrary, therefore I can believe whatever I want." If the choice were really arbitrary, you would be satisfied with the choice being made arbitrarily: by flipping a coin, or calling a random number generator. (It doesn't matter which.) Whatever criterion your brain is using to decide which word or belief you want, is your non-arbitrary reason.

If what you want isn't currently true in reality, maybe there's some action you could take to make it become true. To search for that action, you're going to need accurate beliefs about what reality is currently like. To enlist the help of others in your planning, you're going to need precise terminology to communicate accurate beliefs about what reality is currently like. Even when—especially when—the current reality is inconvenient.

Even when it hurts.

(Oh, and if you're actually trying to optimize other people's models of the world, rather than the world itself—you could just lie, rather than playing clever category-gerrymandering mind games. It would be a lot simpler!)

Imagine that you've had a peculiar job in a peculiar factory for a long time. After many mind-numbing years of sorting bleggs and rubes all day and enduring being trolled by Susan the Senior Sorter and her evil sense of humor, you finally work up the courage to ask Bob the Big Boss for a promotion.

"Sure," Bob says. "Starting tomorrow, you're our new Vice President of Sorting!"

"Wow, this is amazing," you say. "I don't know what to ask first! What will my new responsibilities be?"

"Oh, your responsibilities will be the same: sort bleggs and rubes every Monday through Friday from 9 a.m. to 5 p.m."

You frown. "Okay. But Vice Presidents get paid a lot, right? What will my salary be?"

"Still $9.50 hourly wages, just like now."

You grimace. "O–kay. But Vice Presidents get more authority, right? Will I be someone's boss?"

"No, you'll still report to Susan, just like now."

You snort. "A Vice President, reporting to a mere Senior Sorter?"

"Oh, no," says Bob. "Susan is also getting promoted—to Senior Vice President of Sorting!"

You lose it. "Bob, this is bullshit. When you said I was getting promoted to Vice President, that created a bunch of probabilistic expectations in my mind: you made me anticipate getting new challenges, more money, and more authority, and then you reveal that you're just slapping an inflated title on the same old dead-end job. It's like handing me a blegg, and then saying that it's a rube that just happens to be blue, furry, and egg-shaped ... or telling me you have a dragon in your garage, except that it's an invisible, silent dragon that doesn't breathe. You may think you're being kind to me asking me to believe in an unfalsifiable promotion, but when you replace the symbol with the substance, it's actually just cruel. Stop fucking with my head! ... sir."

Bob looks offended. "This promotion isn't unfalsifiable," he says. "It says, 'Vice President of Sorting' right here on the employee roster. That's an sensory experience that you can make falsifiable predictions about. I'll even get you business cards that say, 'Vice President of Sorting.' That's another falsifiable prediction. Using language in a way you dislike is not lying. The propositions you claim false—about new job tasks, increased pay and authority—is not what the title is meant to convey, and this is known to everyone involved; it is not a secret."

Bob kind of has a point. It's tempting to argue that things like titles and names are part of the map, not the territory. Unless the name is written down. Or spoken aloud (instantiated in sound waves). Or thought about (instantiated in neurons). The map is part of the territory: insisting that the title isn't part of the "job" and therefore violates the maxim that meaningful beliefs must have testable consequences, doesn't quite work. Observing the title on the employee roster indeed tightly constrains your anticipated experience of the title on the business card. So, that's a non-gerrymandered, predictively useful category ... right? What is there for a rationalist to complain about?

To see the problem, we must turn to information theory.

Let's imagine that an abstract Job has four binary properties that can either be high or low—task complexity, pay, authority, and prestige of title—forming a four-dimensional Jobspace. Suppose that two-thirds of Jobs have {complexity: low, pay: low, authority: low, title: low} (which we'll write more briefly as [low, low, low, low]) and the remaining one-third have {complexity: high, pay: high, authority: high, title: high} (which we'll write as [high, high, high, high]).

Task variety and authority are hard to perceive outside of the company, and pay is only negotiated after an offer is made, so people deciding to seek a Job can only make decisions based the Job's title: but that's fine, because in the scenario described, you can infer any of the other properties from the title with certainty. Because the properties are either all low or all high, the joint entropy of title and any other property is going to have the same value as either of the individual property entropies, namely ⅔ log₂ 3/2 + ⅓ log₂ 3 ≈ 0.918 bits.

But since H(pay) = H(title) = H(pay, title), then the mutual information I(pay; title) has the same value, because I(pay; title) = H(pay) + H(title) − H(pay, title) by definition.

Then suppose a lot of companies get Bob's bright idea: half of the Jobs that used to occupy the point [low, low, low, low] in Jobspace, get their title coordinate changed to high. So now one-third of the Jobs are at [low, low, low, low], another third are at [low, low, low, high], and the remaining third are at [high, high, high, high]. What happens to the mutual information I(pay; title)?

I(pay; title) = H(pay) + H(title) − H(pay, title)
= (⅔ log 3/2 + ⅓ log 3) + (⅔ log 3/2 + ⅓ log 3) − 3(⅓ log 3)
= 4/3 log 3/2 + 2/3 log 3 − log 3 ≈ 0.2516 bits.

It went down! Bob and his analogues, having observed that employees and Job-seekers prefer Jobs with high-prestige titles, thought they were being benevolent by making more Jobs have the desired titles. And perhaps they have helped savvy employees who can arbitrage the gap between the new and old worlds by being able to put "Vice President" on their resumés when searching for a new Job.

But from the perspective of people who wanted to use titles as an easily-communicable correlate of the other features of a Job, all that's actually been accomplished is making language less useful.

In view of the preceding discussion, to "37 Ways That Words Can Be Wrong", we might wish to append, "38. Your definition draws a boundary around a cluster in an inappropriately 'thin' subspace of Thingspace that excludes relevant variables, resulting in fallacies of compression."

Miyamoto Musashi is quoted:

The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him.

Similarly, the primary thing when you take a word in your lips is your intention to reflect the territory, whatever the means. Whenever you categorize, label, name, define, or draw boundaries, you must cut through to the correct answer in the same movement. If you think only of categorizing, labeling, naming, defining, or drawing boundaries, you will not be able actually to reflect the territory.

Do not ask whether there's a rule of rationality saying that you shouldn't call dolphins fish. Ask whether dolphins are fish.

And if you speak overmuch of the Way you will not attain it.

(Thanks to Alicorn, Sarah Constantin, Ben Hoffman, Zvi Mowshowitz, Jessica Taylor, and Michael Vassar for feedback.)