An Algorithmic Lucidity

(originally published at Less Wrong)

"Rules" are a critical social technology for helping people live and work together in peace. From the laws passed by legislatures to govern a whole nation, to the bylaws of a neighborhood homeowner association, to the informal household rules of a single family, explicit rules make it clear to everyone what behavior is required and what behavior is forbidden, without otherwise controling every minute detail of everyone's behavior.

When there are clear rules, people don't have to drive themselves crazy contorting themselves into unnatural shapes to satisfy the whims of some distant Authority. All you have to do is make sure to obey the rules. With that taken care of, you can go about living your life the way you see fit, in freedom and dignity. As can be attested in the annals of human experience from the time of Hammurabi into the present day, it mostly works pretty great—at least compared to the alternatives. In summary, rules are good. It's good to have clear rules, and for people to obey the rules.

Normal people understand this pretty well and probably don't need to read a blog post about it, but some people who aren't normal have a theoretical objection. The space of all possible behaviors is unthinkably vast. What if the formidable intelligence of an adversary who hates everything our Society stands for, comes up with a behavior that's really bad but isn't forbidden by any of Society's rules?

The normal person is unfazed by the theoretical objection. If that happens, you could just make a new rule forbidding that behavior, right? How hard could that be?

The people who aren't normal are unimpressed with this reply. They can tell that the normal person doesn't understand the vastness of the space of possible behaviors at all. If you just make a new rule, surely the formidable intelligence of the adversary will contrive some other eldritch behavior that minimizes Society's utility function while complying to the letter of all of Society's rules. The theory of nearest unblocked strategies in the lore of AGI alignment, and the specter of specification gaming in the practice of ML engineering, make it clear that this is so. Thus, rules won't suffice; we need to empower leaders with the Authority to make judgement calls—even to control the minute details of anyone's behavior, if that's what it takes to safeguard Society's Values.

Now me, I'm normal on my mother's side, which puts me in a good position to understand what both parties to the disagreement are saying. And while my full belief-state about related topics in the theory of decision and optimization is nuanced and complex, on the narrow question of what to do about rules in human Society, I think the normal people have it basically right, and the people who aren't normal are being scared of ghosts. Let me explain.

I do not dispute the lore of AGI alignment, nor the practice of ML engineering. But crucially, the purpose of rules in human Society is highly disanalogous to the purpose of a utility or reward function in AI. Rules aren't supposed to express Society's true Values, let alone be a perfect specification robust to nearest unblocked strategies. The Values live in the hearts of Society's individual women and men, to be expressed in the way they go about living their lives the way they see fit, in freedom and dignity. The rules are just there to stop ourselves from trying to kill each other when your freedom and dignity is getting in the way of my freedom and dignity, so that we can focus on creating Value instead of wasting effort trying to kill each other.

Rules are written to ensure conditions conducive to people living their lives in freedom and dignity when those conditions wouldn't obtain in the absence of a rule. Traffic laws make it clear to everyone when it's safe to enter the road. If everyone just entered the road whenever they felt like it, that would be dangerous, and the danger would interfere with people living their lives in freedom and dignity.

The theory of nearest unblocked strategies can be relevant to rules in human Society to the extent that the conditions that a rule is intended to ensure are something that some people oppose either terminally or due to strong instrumental convergence. Income tax laws are passed so that the government will have money to fund police to enforce all the other laws, but that money has to come from somewhere and people really don't like having less money, so they put the full force of their effort and ingenuity into side-stepping the law with clever nearest unblocked strategies: underreporting cash transactions, hiding money in offshore accounts, recategorizing consumption as business expenses, &c.

But more often, the conditions that a rule is intended to ensure aren't something that people terminally or convergently-instrumentally oppose. The rule merely restricts behavior that people would otherwise engage in instrumentally, but not convergently instrumentally: if the rule is in place, they can and will avoid the behavior in order to comply with the rule.

Lead paint is an environmental hazard, so it was banned in 1978. Because of the ban, paint manufacturers stopped making lead paint. The paint manufacturers did not put the full force of their effort and ingenuity into clever nearest unblocked strategies for increasing the amount of lead in the environment, because they're not environmental lead maximizers, which aren't a real thing. The paint manufacturers just wanted to make paint. When there wasn't a rule against it, they used lead carbonate in their paint because it was convenient, but when there was a rule against it, they stopped. The rule worked—without the need for empowering an Authority to make judgement calls controlling the minute details of everyone's behavior. Why wouldn't it?

In some situations, there might be weak instrumental convergence pressures such that the first attempt at making a rule doesn't quite succeed at ensuring the conditions that the rule was meant to ensure. It turns out that, on further consideration, Society doesn't just want to avoid environmental contamination with lead in particular, but all other toxic heavy metals, too, some of which also happen to be convenient for making paint. So paint manufacturers still ended up using mercury in some paints until 1991 when that was banned, too. But once it was banned, they stopped. Why wouldn't they? They're not environmental mercury maximizers, either, which also aren't a real thing.

The work of coming up with rules to ensure socially beneficial outcomes can be frustrating, because you won't always get the rules exactly right the first time. You might need to iterate. But it's a finite and achievable amount of work, not an unwinnable unending battle against the formidable intelligence of an adversary who hates everything your Society stands for, because those mostly aren't a real thing either.

In conclusion, I think that people who think rules are unworkable and instead want to empower an Authority to make judgement calls controlling the minute details of everyone's behavior need to read less science fiction and spend more time relating to other people in their Society as people. Notwithstanding that terrifying alien superintelligences couldn't be constrained by rules because a merely human intellect lacks the capabilities to enumerate all the nearest unblocked strategies, other people in your Society are not terrifying alien superintelligences. We're just people who don't have exactly the same preferences as you. We won't always agree, but it shouldn't be this hard to live in peace with each other. If there are problems, you can just make a new rule!

(Thanks to Robert Mushkatblat and Ben Pace.)

(originally published at Less Wrong)

One of the most underrated essays in the post-Sequences era of Eliezer Yudkowsky's corpus is "Four Layers of Intellectual Conversation". The degree to which this piece of wisdom has fallen into tragic neglect in these dark ages of the 2020s may be related to its ephemeral form of publication: it was originally posted as a status update on Yudkowsky's Facebook account on 20 December 2016 and subsequently mirrored on Alyssa Vance's The Rationalist Conspiracy blog, which has since gone offline. (The first link in this paragraph is to an archive of the Rationalist Conspiracy post.)

In the post, Yudkowsky argues that a structure of intellectual value necessarily requires four layers of conversation: thesis, critique, response, and counter-response (which Yudkowsky indexes from zero as layers 0, 1, 2, and 3).

The importance of critique is already widespread common wisdom: if a thesis is advanced and promulgated without any serious effort to examine why it might be in error, then it likely is in error, both because it can't have incorporated corrections from critiques (which are ex hypothesi absent) and because the author lacks incentives to offer a correct thesis in the first place: if being right is difficult and there's no social penalty for being wrong, then most humans will inexorably find themselves on the easy course of being wrong even without any conscious intent to deceive. That is, in the words of the post, the problem with "a conversation consisting of people saying X and nobody saying 'hey maybe not-X'" is that "people could say stupid things about X, and nobody would call them on the stupidity." Yudkowsky aptly concludes: "Yikes!"

Yudkowsky's key observation going beyond common wisdom is that the necessity of social incentives to be correct also applies to the level-1 critique and level-2 response, not just the level-0 thesis—and moreover, that the higher levels are critical for the lower levels to maintain their force. The mere existence of level-1 critics won't suffice to keep level-0 thesis-proposers on their toes, if the level-1 critics are themselves not on their toes because they don't anticipate being held to account by level-2 responses. Likewise, level-2 responses won't suffice to keep level-1 critics on their toes if the level-2 responders don't anticipate being held to account by level-3 counter-responses. Without all four levels, the whole structure comes apart.

Yudkowsky offers public debates about evolution and molecular nanotechnology as examples of discourses with a missing level 3. If biologists explain evolution (level-0 thesis), religious scholars insist that God must have started it all (level-1 critique), biologists explain leading theories of abiogenesis (level-2 response), but religious scholars don't engage with the abiogenesis work, then the conversation has failed to secure a level-3 counter-response.

It matters that the higher levels are being held to a high enough standard that people would lose face if they played dumb. If K. Eric Drexler writes technical books and papers about the possibilities of nanotechnology (level-0 thesis), Richard Smalley objects that manipulator arms themselves made of atoms would be too "fat" and "sticky" to work as a molecular assembler and that this problem is fundamentally uncircumventable (level-1 critique), Drexler et al. reply that biological ribosomes demonstrate that the problem is not fundamentally uncircumventable even though Drexler's proposals have a "mechanical" rather than "biological" character (level-2 response), and Smalley objects that biological systems can't work with the materials used in technology and that Drexler has departed from real chemistry (level-3 counter-response), then all four levels are formally present, but one is left with disquieting sense that the level-3 counter-response has failed to truly connect with the level-2 response. (Drexler et al.'s level-2 response had brought up biology as an existence proof that the "fat finger" problem didn't sink the entire idea of nanotechnology; pointing out that biology can't do the things that Drexler had conjectured nanotechnology could, would seem to be missing the point.)

Yudkowsky laments that the academic journal system, with the possible exception of analytic philosophy, mostly only canonizes levels 0–2: it's uncommon to see a journal article that's a reply to a reply to a reply to another. To the extent that real intellectual progress is being made in most fields, the real work is probably happening at conferences or on email lists, with the journals merely recording the work after the fact. Yudkowsky sings the praises of transhumanist mailing lists of the late '90s, where people who might otherwise succumb to the temptation to play dumb were kept in check for fear of Robin Hanson's clinically precise rebuttals. Nick Bostrom's 2014 Superintelligence merely packaged up for the public the outcome of a hard-fought discourse that had occurred elsewhere.

A shortcoming of the original post is a lack of concrete examples (with labeled levels) of the four levels of conversation succeeding rather than failing. (We didn't get much detail about exactly what happened on that mailing list.)

The impact of the replication crisis on the study of priming effects might be a candidate. In 1996's "Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action", John A. Bargh and collaborators reported that college students directed to solve a puzzle involving words related to elderly people walked slower when leaving the lab (level-0 thesis). Sixteen years later, in "Behavioral Priming: It's All in the Mind, but Whose Mind?", Stéphane Doyen et al. ran a replication that failed to reproduce the original result on walking speed when the experimenter administering the puzzle was blinded to the hypothesis being tested, but did reproduce the result when the experimenter was led to believe that there would be a priming effect (level-1 critique). Bargh wrote a blog post, "Nothing in Their Heads", arguing that the experimenter was blinded in the original 1996 study, that Doyen et al. over-primed with too many elderliness-related words (which Bargh argued could destroy the effect), and that Doyen et al. didn't check if subjects had slowness-related stereotypes about the elderly (level-2 response). Though the original post's comment section seems to have been lost to history, science journalist Ed Yong documented responses to Bargh's post by commenters on the post and by coauthors of Doyen et al., claiming inaccuracies in the post, and that, in any case, a truly robust priming effect wouldn't be so fragile to such small changes in the study design (level-3 counter-response).

Nor did the conversation about this particular paper drop silently into the void: soon, the famed Daniel Kahneman would write a letter to priming research practitioners named to him by Bargh on bringing more rigorous study designs to the field, which has continued to be plagued by replication difficulties. The discussion made an impact on Society's collective beliefs. The attempt at discourse was more than a noble gesture. It hadn't all been for nothing.

A natural question to ask about the four-levels framework is: why four levels, specifically? Doesn't the recursion of level n needing level n + 1 go off to infinity?

The original post leaves the question unanswered, but a potential answer can be found in Yudkowsky's tongue-in-cheek Law of Ultrafinite Recursion, which states that, in practice, infinite recursions are at most three levels deep. The Law of Ultrafinite Recursion is deliberately silly if construed as a literal claim about computer science but is surprisingly fruitful as a claim about human psychology: it's pretty natural to ask what Alice thinks that Bob thinks about Carol, but asking what Alice thinks that Bob thinks that Carol thinks about Dave feels like a stretch.

If the limited human grasp of recursion rounds "four" up to "infinity", then the chain of thesis–critique–response–counter-response is enough to establish the expectation of unlimited-depth accountability and remove the incentive to bluff. A different species with greater working memory capacity, whose members could follow a backwards induction farther, might need more counter-counter-responses and counter-counter-counter-responses to experience the same salutary effect.

The four-levels model is about robust disagreements, which are usually pretty frustrating for all involved. No one likes being told they're wrong, especially by people who (so it always seems from the other side) are themselves obviously wrong.

The frustration is not optional. The recursive pressure forcing you to come up with your best arguments and responses to counter the adversary's critiques and counter-responses only works if the adversary is allowed to be frustrating; it's not their job to make it easy for you. Equivalently, it's not your job to make it easy for them. Only by facing this test can your combined efforts build an intellectual edifice guided by the beauty of your weapons.

Bargh's blog post complains that "oddly for an article that purported to fail to replicate one of [his] past studies", he wasn't asked to review Doyen et al. But it's not odd: journals generally want reviewers to be independent. For example, the International Committee of Medical Journal Editors recommends that peer reviewers should "declare their relationships and activities that might bias their evaluation of a manuscript and recuse themselves from the peer-review process if a conflict exists."

If Bargh were the one who got to decide who is allowed to speak on the record about potential flaws in Bargh et al. 1996, then Society would lose out on its chance to determine whether Bargh et al. 1996 is actually correct. Any single conversational locus that forgets or denies this obvious principle is at serious risk of degenerating into an echo chamber if it hasn't already.

(originally published at Less Wrong)

The Value of a Comment Is Determined by Its Text, Not Its Authorship

I sometimes see people express disapproval of critical blog comments by commenters who don't write many blog posts of their own. Such meta-criticism is not infrequently couched in terms of metaphors to some non-blogging domain. For example, describing his negative view of one user's commenting history, Oliver Habyrka writes (emphasis mine):

The situation seems more similar to having a competitive team where anyone gets screamed at for basically any motion, with a coach who doesn't themselves perform the sport, but just complaints [sic] in long tirades any time anyone does anything, making references to methods of practice and training long-outdated, with a constant air of superiority.

In a similar vein, Duncan Sabien writes (emphasis mine):

There's only so much withering critique a given builder is interested in receiving (frequently from those who do not themselves even build!) before eventually they will either stop building entirely, or leave to go somewhere where buildery is appreciated, rewarded, and (importantly) defended.

I find this stance deeply puzzling. In general, the value of a critical blog comment is in potentially alerting readers to an error, omission, or other shortcoming of the post. (If the alleged shortcoming does not in fact exist, the value of the comment is negative.) This value clearly does not depend on the identity of the author!

I recently committed the sin of publishing a post which suffered from multiple shortcomings. For one, I implied that the set of continuous functions from ℝ to ℝ equipped with the uniform norm is a normed space.

That was wrong of me. The thing I wrote was wrong. The reason that the thing I wrote was wrong is because norms are defined as functions that output a real number, but there exist continuous functions that are unbounded, and if we attempt to take the uniform norm of such a function—the least upper bound of its absolute value—we get +∞, which isn't a real number. (In contrast, the space of continuous functions from a compact domain to ℝ under the uniform norm is a normed space, because by the extreme value theorem, those functions are bounded.)

A comment pointed out that I was wrong. That comment was valuable because it alerted readers of the comment section to an error in the post. (It also happened to alert me, the author, because I happened to be one of the readers of the comment section.)

The reason it makes sense for me to write "A comment pointed out that I was wrong" even though comments aren't people is because the identity of the commenter doesn't matter. It doesn't matter what their name is. It doesn't matter whether they have a math degree. It doesn't matter whether they went to school at all.

It doesn't matter whether they're human. If a large language model had written the same comment, it would be the same comment. The same sequence of bytes would be stored in the content field of the Comments table of the website's database. Because it would be the same sequence of bytes, the effect of rendering those bytes as text on a monitor and showing them to a human would be the same. The human reading the comment has no way of knowing who or what wrote those bytes to the database. In the language of causal graphical models, we can say that the text of the comment "screens off" the process that produced it.

In principle, it doesn't matter whether the process that generated the comment is "intelligent" in any sense. A so-called "large language model" is just a conditional probability distribution expressed as a computer program: generating text is sampling from the distribution. But you could do that with any distribution. If by some exponentially improbable cosmic coincidence, uniformly sampling from printable ASCII characters (in Python, ''.join(chr(random.randint(32, 126)) for _ in range(n)) for a sample n characters long) somehow produced the same comment, it would still be the same comment.

Given that a commenter's name, educational attainment, humanity, or existence as an independent entity does not affect the value of a given comment, it should be clear that another thing that doesn't matter is whether the commenter writes blog posts in addition to blog comments. That doesn't matter. Why would someone think that matters?

However, Critic Contributions Can Inform Uncertain Estimates of Comment Value

Except we should not be premature. The people who write metaphors about coaches who don't themselves perform the sport they coach or builders who do not themselves build, seem to think it matters. We should search harder for reasons why someone would think that.

It turns out that there are some important nuances here that must be addressed. The value of a comment doesn't depend on whether the commenter also writes posts—if the value of the comment is known with certainty (such that its authorship is screened off). If we're uncertain about the comment's value, our uncertain estimate of its value can depend on what other things the author has done. In Bayesian terms, the likelihood provided by our imperfect estimation of the comment's value isn't strong enough to fully overcome our author-based prior.

Author-based priors can be decision-relevant, as can be seen from the limiting case of the uniform printable ASCII distribution: you wouldn't want to give a random-character-generating program commenting privileges on your blog, because an exponentially vast hypermajority of its output is worthless gibberish (and of the tiny fraction that looks sensible by sheer cosmic coincidence, the vast hypermajority won't furthermore happen to be right by another cosmic coincidence). Even July 2025–era language models don't make the cut in most blog administrators' eyes.

The decision-relevance of author-based priors neatly explains the appeal of the coach and builder metaphors. If aspiring athletes and builders don't know how to distinguish between good and bad advice (and ignore the bad advice at zero cost), it makes sense for them to only listen to people likely on priors to give good advice, which would mostly be people who have excelled at the activity before. Taken on their own terms, the examples make sense: you probably wouldn't want a coach who had never been a player, a building advisor who had never built.

There's still a problem, however: just because the examples make sense on their own terms, doesn't mean they make sense as blogging analogies. It makes sense that a coach who had never played would thereby be a bad coach, because the way you gain intimate knowledge of the best way to play the game is by playing it for years.

But would a commenter who had never written "top-level" posts thereby be a worse commenter? It's hard to see why that would be the case. In the analogy, coaching is an activity that depends on playing, but comment-writing doesn't seem to depend on post-writing to nearly the same extent or even in the same way, in large part because it's not even clear to what extent comment-writing and post-writing are even different activities, rather than just being the same activity, writing. (It's not uncommon that text that was originally drafted with the intent of being a "comment", ends up being revised into a "post.")

Maybe if a post is on some specialized topic, like DNA polymerase mutations in C. elegans or maritime salvage law in international waters, it might make sense to disapprove of ignorant commenters mouthing off without themselves being nematode microbiologists or navy JAGs. It's not crazy to think that people who aren't nematode microbiologists won't have any good opinions about DNA polymerase mutations in C. elegans, such that we're not missing anything important by refusing to let them comment.

But it doesn't make sense to gatekeep blog commenting privileges on writing posts for the same blog, because there's no particular reason why someone shouldn't happen to do more of their writing in the form of comments rather than posts. That doesn't matter. Why would someone think that matters?

A Caveat: Critic Contributions Can Be Relevant If You Don't Care About Maximizing Correctness

That wasn't a rhetorical question. Why would someone think that matters? The explanations given above for why the value of a critical comment doesn't depend on its author, and why whether a commenter also writes posts does not have much evidential bearing on the uncertain value of a comment, seem pretty straightforward, even obvious. Where is the error in the reasoning?

If there's no error in the reasoning, perhaps the disagreement comes down to different starting premises. It doesn't matter whether a commenter also writes posts—if one accepts as a premise that the value of a critical blog comment is in potentially alerting readers to an error, omission, or other shortcoming of the post. If one denies that premise and embraces some other theory of comment value, other conclusions are possible.

For a simple example of what such an alternative theory could look like, one could hold that the function of a critical blog comment is to attempt to raise the commenter's social status and lower the status of the post author. Then, given some separate criterion of who deserves what status, a good comment would be by someone who deserves to be high status, criticizing a post written by someone who deserves to be low status. Conversely, a bad comment would be by someone who deserves to have low status, criticizing a post written by someone who deserves to have high status—and the more persuasive the comment is, the worse it is, because more successful persuasion increases the misallocation of status (in the minds of persuaded readers) to the commenter who, ex hypothesi, doesn't deserve it.

Of course, that's not the only possible alternative theory of comment value. One could imagine an intricate "hybrid" theory that strikes a carefully computed compromise between alerting readers to errors and omissions in a post, and optimizing status allocation with respect to some criterion of deservingness.

Suppose the administrators of some website are trying to optimize some quantity, like "total number of interesting ideas posted to the website", or maybe "advertising revenue." Let's go with ad revenue because it's easier to measure and should be a good proxy for interesting ideas. (If the website is the place to go for interesting ideas, then lots of people will want to visit it, and advertisers will pay for all those people's clicks.) Suppose furthermore that contributors are motivated by status: if people lose too much status from their posts or comments, they'll stop writing, which has a negative effect on ad revenue.

Under this hybrid theory of comment value, it can make sense to disapprove of people who write critical comments and not posts, if the error-correction value of the comments is outweighed by lost ad revenue due to demotivated authors.

Thus, our earlier conclusion must be revised to be conditional. It doesn't make sense to disapprove of commenters who don't write posts, if you only care about correctness. If you care about something other than correctness, such as ad revenue, then it can make sense to disapprove of commenters who don't write posts. The inference also works in the other direction: if you disapprove of commenters who don't write posts, that implies that you care about something other than correctness.