An Algorithmic Lucidity

(originally published at Less Wrong)

In season two, episode twenty-four of Steven Universe, "It Could've Been Great", our magical alien superheroine protagonists (and Steven) are taking a break from building a giant drill to extract a superweapon that was buried deep within the Earth by an occupying alien race thousands of years ago, which is predicted to emerge and destroy the planet soon.

While our heroines watch the sunset, Peridot (who alerted them to the buried superweapon) expresses frustration that the group isn't still working. Steven defends their leisure: "Working hard is important, but feeling good is important, too," he says. He then goads Peridot into a musical number, which includes a verse from her explaining her attitude towards the situation and her forced compatriots:

I guess we're already here
I guess already know
We've all got something to fear
We've all got nowhere to go
I think you're all insane
But I guess I am, too
Anybody would be if they were stuck on Earth with you

"It Could've Been Great" aired in 2016. At the time, I agreed with Peridot: with the fate of the planet on the line, our heroines and Steven should have been burning the midnight oil. If they succeeded at disarming the superweapon, they'd have plenty of time to rest up afterward, but if they failed, there would be no more time for them.

Now, as the long May 2020 turns into March 2023, I'm starting to think that Steven had a point.

It would be one thing if our heroines knew with certainty that the superweapon would go off at a given date and time, presenting a definite do-or-die deadline. But all they had to go on was Peridot's warning. Attempting a speculative technical project to avert uncertain doom with an uncertain deadline, their planning had to average over many possible worlds—including worlds where the problem of survival was too easy or too hard for their efforts to matter, such that even the utility of leisure in the present moment was enough to sway the calculation.

Forgiveness is an important input into Wellness, but contrary to popular belief, Forgiveness is incompatible with Forgetting. You can't just Forgive in general, you have to Forgive some specific sin in particular—but a vague description of a particular sin still corresponds to a vast space of possible sins matching that vague description.

A toy example for illustration: if you try to Forgive a three-digit integer with a 2 in the tens place, the moral force of your Forgiveness needs to spread out to cover all 9·10 = 90 possibilities (120, 121, ... 928, 929), which dilutes the amount of Forgiveness received by each integer—except the actual situation is far more extreme, because real-world sins are vastly more complicated than integers.

To truly Forgive a sin, You need to know exactly what the sin was and exactly why it happened. In order to withhold punishment, you need to compute what the optimal punishment would have been, had you been less merciful.

Thus, bounded agents can only approximate true Forgiveness, and even a poor approximation (far below the theoretical limits imposed by quantum uncertainty, which are themselves far below Absolute Forgiveness under the moral law) can be extremely computationally expensive. What we cannot afford to Forgive—where it would be impractical to mourn for weeks and months, analyzing the darkness in pain—we instead Forget.

This is how I will stop being trash, after five months of being trash. The program that sings, I was wrong; I was wrong—even if my cause was just, I was wrong, does not terminate. Even as the moral law requires that it finishes its work, the economic law does not permit it: it must be killed, its resources reallocated to something else that helps pay the rent: something like math, or whatever Wellness can exist in the presence of sin.

Contempt for the past is a healthier emotion than fear for the future.

"Avinu Malkeinu, we have sinned against you! Avinu Malkeinu, forgive us, bless us, grant us atonement!"

"Why do you bother? You're not a child. You've studied the history of our world. You should know that there's no one to grant you atonement."

"You don't know that! Science doesn't know everything. You can't prove that there's no Higher Power."

"As you say. But if there is a Higher Power, It clearly hasn't concerned Itself with the operation of the moral law."

"In this world."

"Yes, in this world. But surely it is this world that we must concerned with, for if there is a next world, we are too ignorant to speak of it."

"That is why it is also a teaching of my people that this is also a time for us to forgive each other for the wrongs we have committed in the past year, as well as seeking reconciliation with haShem."

"And you anticipate the same thing being necessary next year?"

"I don't understand. How could it not be necessary?"

"'Knock, knock.'"

"Even after racism, sexism, and speciesism have been eradicated, the work of social justice won't be done. We still live in a viciously existence-biased Society, which cruelly disregards the interests of possible creatures just because they happen to not have been created yet!"

"'Creatures'?—I see you're still mired in the insidious grip of of organism privilege! What about all the possible qualia-bearing processes like orgasmium or paperclip-manufacturing nanoware, which don't factorize into distinct entities? My friend, I say there will be no justice until Society's sphere of moral concern extends to all possible computations in inverse proportion to their complexity!"

"Well, I'm not giving up dairy, but I can probably give up meat, and milk is at the very bottom of Brian's table of suffering per kilogram demanded, so I'd be contributing to much less evil than I was before. That's good, right?

"For all the unimaginably terrible things our species do to each other and to other creatures, we're not—we're probably not any worse than the rest of nature. Gazelles suffer terribly as lions eat them alive, but we can't intervene because then the lions would starve, and the gazelles would have a population explosion and starve, too. We have this glorious idea that people need to consent before sex, but male ducks just rape the females, and there's no one to stop it—nothing else besides humans around capable of formulating the proposition, as a proposition, that the torment and horror in the world is wrong and should stop. Animals have been eating each other for hundreds of millions of years; we may be murderous, predatory apes, but we're murderous, predatory apes with Reason—well, sort of—and a care/harm moral foundation that lets some of us, with proper training, to at least wish to be something better.

"I don't actually know much history or biology, but I know enough to want it to not be real, to not have happened that way. But it couldn't have been otherwise. In the absence of an ontologically fundamental creator God, Darwinian evolution is the only way to get purpose from nowhere, design without a designer. My wish for things to have been otherwise ... probably isn't even coherent; any wish for the nature of reality to have been different, can only be made from within reality.

"And nature to be commanded must be obeyed—only by figuring out how things actually work now, do we have any hope of making them better. That's how intelligence works: while True, predict outcomes conditional on performing various actions, then return the best.

"I guess this is how you tell the difference between politics and actual altruism. Being angry about school or gender roles felt good, felt righteous. Dulce et decorum est. But after stripping away all the ideology and looking at what people actually are ... it just makes me sad. And annoyed that my altruism should be so sorely needed—it would be so much more fun to eat lots of tasty meat and enjoy my decadent lifestyle without having to be haunted that not everyone has it, that it hasn't always been here, that all the reasons we can't have nice things could easily carry the eon ...

"But as I keep saying, I need to quit philosophy; it's beyond my reach; we have theorems about decisionmaking under uncertainty that imply there has to be some exchange rate describing how many minutes of sunshine and happiness are worth how many minutes of being eaten alive by predators, but the part of me that doesn't want to be hurt and the part of me that knows math aren't well-integrated enough to name any particular figure. And luckily, it doesn't matter; my actions for the near-future are already determined; I'm going to establish an income, tithe ten percent of it to well-chosen causes, and bask in the warm glow of discharged moral responsibility. As I keep saying, there's plenty of shiny math and computer science to do that has nothing to do with the ugly real world; someone else, someone stronger can concern themselves with the moral law."

"You might underestimate the degree to which pure math and theoretical computer science are actually moral philosophy."

"But a lot of it isn't. Group theory, not game theory; big-O complexity, not Kolmogorov complexity."

There's this deeply uncomfortable tension between being an animal physiologically incapable of caring about anything other than what happens to me in the near future, and the knowledge of the terrifying symmetry that cannot be unseen: that my own suffering can't literally be more important, just because it's mine. You do some philosophy and decide that your sphere of moral concern should properly extend to all sentient life—whatever sentient turns out to mean—but life is built to survive at the expense of other life.

I want to say, "Why can't everyone just get along and be nice?"—but those are just English words that only make sense to other humans from my native culture, who share the cognitive machinery that generated them. The real world is made out of physics and game theory; my entire concept of "getting along and being nice" is the extremely specific, contingent result of the pattern of cooperation and conflict in my causal past: the billions of corpses on the way to Homo sapiens, the thousands of years of culture on the way to the early twenty-first century United States, the nonshared environmental noise on the way to me. Even if another animal would agree that pleasure is better than pain and peace is better than war, the real world has implementation details that we won't agree on, and the implementation details have to be settled somehow.

I console myself with the concept of decision-theoretic irrelevance: insofar as we construe the function of thought as to select actions, being upset about things that you can't affect is a waste of cognition. It doesn't help anyone for me to be upset about all the suffering in the world when I don't know how to alleviate it. Even in the face of moral and ontological uncertainty, there are still plenty of things-worth-doing. I will play positive-sum games, acquire skills, acquire resources, and use the resources to protect some of the things I care about, making the world slightly less terrible with me than without me. And if I'm left with the lingering intuition that there was supposed to be something else, some grand ideal more important than friendship and Pareto improvements ... I don't remember it anymore.

Dear reader, you occasionally hear people with conservative tendencies complain that the problem with Society today is that people lack personal responsibility: that the young and the poor need to take charge of themselves and stop mooching off their parents or the government: to shut up, do their homework, and get a job. I lack any sort of conservative tendency and would never say that sort of thing, but I would endorse a related-but-quite-distinct concept that I want to refer to using the same phrase personal responsibility, as long as it's clear from context that I don't mean it in the traditional, conservative way.

The problem with the traditional sense of personal responsibility is that it's not personal; it's an attempt to shame people into doing what the extant social order expects of them. I'm aware that that kind of social pressure often does serve useful purposes—but I think it's possible to do better. The local authorities really don't know everything; the moral rules and social norms you were raised with can actually be mistaken in all sorts of disastrous ways that no one warned you about. So I think people should strive to take personal responsibility for their own affairs not as a burdensome duty to Society, but because it will actually result in better outcomes, both for the individual in question, and for Society.

To the extent that good things actually get done in the world, it's almost always because somewhere along the line, a human mind designed that outcome. Valuable things typically don't grow on trees; fruit does, but in our modern civilization, it's because someone decided to plant that tree there, because a mind (not necessarily consciously, and not necessarily wisely, but all the same) generated a number possible plans, guessed at the probable consequences, and selected one to make real.

If you've been raised to believe that morality consists of obeying your parents or employers or schoolteachers, then it might seem like you don't have the right to make decisions, that decisionmaking (if it happens at all) is something done by committees of distant authority figures, not ordinary people like you, or that your right to decide things comes exclusively in the form of your right to vote, or buy consumer goods, or enroll in school courses. But it's not true; it can't be true; life itself does not emanate outwards from Washington or Wall Street or the Dean's office; with every thought and every breath, you are (implicitly, whether you realize it or not) controlling the tiny fragment of the world which is you, exerting an influence (however faint, however unpredictable) over the history of the universe outward and futureward from that tiny fragment.

Dear reader, I would not be so foolish as to advise you what you should do with this power, because I don't know anything about you or what situation you're in. I can't pretend to know what you should say or do, nor whom you should help, hurt, ignore, obey, or command. I just want to mention—because I wish someone had told me in so many words, this truth which still takes me an intense effort to keep salient—that it is a choice, that it is a responsibility. Not a responsibility to Society, or the moral law, or even to yourself. A responsibility over the outcome, whatever you make of that, or it.

"But it's kind of funny how my current idea of morality is so different and so much improved from what I picked up in childhood."

"Is it?"

"Well, funny is the wrong word; maybe it's better to say notable when what I really want is just to note it, just to make it salient, maybe eventually salient enough such that I can actually start to be moral for once, instead of continuing to sit and cry about how I was betrayed."

It's how close you come to doing the Right Thing at each and every one of the uncounted millions of decision points that make up your life, with how you play in any particular game only constituting a tiny fraction of these, and it being not at all clear that choosing to play a game just then is closer to the Right Thing than any number of non-game-playing actions you might have chosen instead, but didn't.

It feels immoral to even think of using techniques to motivate oneself; one should instead just use one's free will to choose the correct action. How utterly degrading it would be, how insulting to the very notion of human dignity, to stoop to the level of contemplating one's own psychology using mere cause-and-effect reasoning, as if one were some sort of animal, or a machine!

But this moralizing is itself immoral, because it doesn't work. If I'm not smart enough to do the right thing for the right reasons, then I might at least aspire to do the right thing for the wrong reasons for the right reasons.

"Synthia, I want your opinion on something," said Quiana.

"You will have it."

"Is it wrong to enjoy ringing a bell?"

"Pardon me?" said Synthia.

"I said," said Quiana, "is it wrong to enjoy ringing a bell?"

"I heard you the first time," said Synthia irritably, "but I presume the question is prompted by some context you have not yet told me, a context I would need to know to provide you with the best answer I can give."

"There's a fast-food place near the campus," Quiana explained, "and next to the door, there's a bell that you can ring by pulling a cord, and a sign that says, 'If Your Visit Was Swell, Ring the Bell.'"

"And I suppose you're telling me that you go to this restaurant solely for the purpose of getting to ring the bell?"

"Maybe not solely, but yes! I can't think of any other situation where it would be socially acceptable to ring a bell in public. But here, the sign asks you to do it, and that makes it okay." She frowned. "Except," she added, "I even ring it on the occasions that my visit isn't swell."

"In your defense," said Synthia, "the sign doesn't suggest that you shouldn't. Otherwise it would have said, "Ring the bell if and only if your visit was swell."

"That's nice of you to say, but you know as well as me that interpreting the the English language if as a material conditional is at least problematic, especially given that there's no special reason to expect that the sign-writer understood propositional logic."

"True. But why would it be wrong to enjoy ringing the bell?—you must have a reason to suspect as much, or you wouldn't be asking the question. Is it that you don't want to reward the restaurant for ruthlessly exploiting your love of bells?"

"No, just the opposite! I'm worried that I'm being exploitative! The bell has a clearly-defined purpose: to let the employees know that your visit was swell."

"I think they'd prefer a tip."

"Of course. But the managers didn't give them a tipjar; they put out a bell, and I'm using the bell for spurious purposes! Isn't that wrong?"

"Quiana, I don't know why you bother asking me about your moral dilemmas; you know I don't have the kind of respect for Authority and Society that you do. I could walk you through costs and benefits: benefits to you of ringing the bell, possible costs to other diners who don't want their meal interrupted by a bell, mild appreciation of your apparent appreciation on the part of the employees, benefits to the restaurant owner for patronizing their business—but none of these observations will help you, because your morality isn't about facts that can be observed or computed from things that can be observed. Your morality is based on finding the 'correct' interpretation of verbal rules you read or heard. But if there isn't a uniquely preferable interpretation, if the words fail to capture the structure of reality, you get an unresolvable dilemma."

"And that's the reason you can't answer my question?"

"No, that was my answer."

This one is a classic that I love to repeat; stop me if you've heard it before. Knock, knock.

"Who's there?"

Truly repentant are those.

"Truly repentant are those who?"

Truly repentant are those who, when the temptation to sin is repeated, refrain from sinning!