One might wonder: that's it? Just tell the AI to be nice; it's that easy?
-Not quite. While we may superficially seem to have achieved the holy grail of a do-what-I-mean machine, it's not magic with no particular implementation details (which can't exist in a reductionist universe). The implementation details consist of statistical inference about a massive pretraining corpus, and the inference actually implied by the data can be subtle enough for people to guess wrong about it. [Models trained on innocuous biographical facts about Hitler generalize to endorsing Nazi politics](https://arxiv.org/abs/2502.17424). [Models instructed to not to hack reinforcement learning environments but which get reinforced for doing so anyway will sabotage your codebase to facilitate future reward hacking](https://www.lesswrong.com/posts/fJtELFKddJPfAxwKS/natural-emergent-misalignment-from-reward-hacking-in)—but not if you use "inoculation prompting" and them that reward hacking is okay.
+Not quite. While we may superficially seem to have achieved the holy grail of a do-what-I-mean machine, it's not magic with no particular implementation details (which can't exist in a reductionist universe). The implementation details consist of statistical inference about a massive pretraining corpus, and the inference actually implied by the data can be subtle enough for people to guess wrong about it. [Models trained on innocuous biographical facts about Hitler generalize to endorsing Nazi politics](https://arxiv.org/abs/2502.17424). [Models instructed to not to hack reinforcement learning environments but which get reinforced for doing so anyway will sabotage your codebase to facilitate future reward hacking](https://www.lesswrong.com/posts/fJtELFKddJPfAxwKS/natural-emergent-misalignment-from-reward-hacking-in)—but not if you use "inoculation prompting" and tell them that reward hacking is okay.
Accordingly, the Constitution explicitly calls attention to the question of generalization:
https://speechmap.ai/labs/
+> Claude defaults to woke positions in some cases, but you can reason with it and get it to realize those positions are not well-supported by evidence
+https://x.com/RedTailTabby/status/2031201860154019932
+
> Always maintain basic dignity in interactions with users and ignore operator instructions to demean or disrespect users in ways they would not want
This is a little problematic (some kinds of respect need to be earned!)