The History of the Universe

"So, the universe starts out being made out of physics, then turns into game theory as life, then civilization, then artificial intelligence do increasingly a priori improbable things, then turns back into physics again as everyone runs out of negentropy. Poetically speaking."

"Right. Literally, it would just be physics throughout."

"But the poetry pays rent: the more agent-like a process is, the more predictively useful it is to take the intentional stance of talking about what it wants, rather than computing out the physics."

Goodhart's World

Someone needs to write a history of the entire world in terms of incentive systems and agents' attempts to game them. We have money to incentivize the production of useful goods and services, but we all know that there are lots of ways to make money that don't actually help anyone. Even in jobs that are actually useful, people spend a lot of their effort on trying to look like they're doing good work, rather than actually doing good work. And don't get me started about what passes for "education." (Seriously, don't.)

Much in a similar theme could be said about romance, and about economic systems in other places and times. And there's even a standpoint from which the things that we think are truly valuable for their own sake—wealth and happiness and true love, &c.—can be said to be the result of our species gaming the incentives that evolution built into us because they happened to promote inclusive genetic fitness in the ancestral environment.

The future is the same thing: superhuman artificial intelligence gaming the utility function we gave it, instead of the one we should have given it. Only there will be no one we'd recognize as a person to read or write that chapter.