Expected Value Maximization in Extreme Cases

The decision theory framework developed by John von Neumann and Oskar Morgenstern gives a way, under particular conditions, to pick the better of two actions with uncertain outcomes by evaluating the “expected value” of each action, and picking the one with the higher EV. In particular, suppose we have a set $ Z $ of outcomes, and a set $P$ of probability functions $p: Z \to [0,1]$. Here, elements of $P$ represent actions. If I take the action associated with $p \in P$ , I don’t know for sure what outcome it will lead to, but I do know that the probability it leads to any outcome $z$ is $p(z)$. When we have such sets $Z$ and $P$, the von Neumann-Morgenstern Representation Theorem tells us that a preference relation $\le$ defined on $P$ satisfying the Substitution and Archimedean Axioms (defined later) is necessary and sufficient for the existence of a utility function $u: Z \to \mathbb{R}$ such that \[ p \le q \text{ if and only if } \sum_{z \in Z}p(z)u(z) \le \sum_{z \in Z}q(z)u(z). \] (Note that the former $\le$ is the preference relation, while the latter $\le$ is the total order relation defined on $\mathbb{R}$.) That is, if we have probability functions $p$ and $q$, i.e. actions, and we have a preference relation defined on our set of actions that satisfies the two axioms, we can figure out what the preference relation says we should do by figuring out the appropriate utility function $u$, and then computing the expected values of each probability function with respect to that utility function. We will write $\mathbb{E}(p(x))$ to mean the expected utility $\sum_{z \in Z}p(z)u(z)$.

Implicit in the von Neumann-Morgenstern framework, or at least in applications of the theory, is that the conditions imposed on the preference relation are conditions of rationality, that it’s reasonable to assume that a rational agent’s preferences adhere to the Substitution and Archimedean Axioms. If we are to make decisions based on expected utility calculations, we should think that the conditions under which our preferences align with these calculations are rational. More broadly, to apply this decision-making framework of expected value maximization, we would hope that if $\mathbb{E}(p(x)) > \mathbb{E}(q(x))$, action $p$ is more appealing to us than action $q$. If not, either there's an irrationality in our preferences, or we think that either the Substitution or Archimedean Axiom is irrational. This paper will explore two cases in which expected value maximization seems bad, and argue that this is evidence that the Archimedean Axiom is doing more than just enforcing rationality. It will then consider the ethical notion of utility in light of this idea that it's unclear whether utility is relevant to decision-making at all.

Consider first the case of Pascal's Mugging, a hypothetical proposed by Eliezer Yudkowsky and with a variation by Nick Bostrom. A mugger demands five dollars from you, but he doesn't have a gun. Instead, he claims that you live in the Matrix, he's from outside the matrix, and if you don't give him the money, he's going to go back outside the matrix, generate a simulation with a VERY LARGE number of people, and painfully kill them all. The mugger knows that you, Pascal, are a benevolent person: you have lower utilities for outcomes where people die than outcomes where they don't, and lower utilities for outcomes where more people die than where fewer people do. He also knows that you're an expected utility maximizer. You tell him it's so unlikely that he's telling the truth that despite the immense scale of what he's threatening, there's more EV in taking the risk and keeping the $5. So he squares the number of people he threatens to kill. Surely, he argues, the probability that he's from outside the matrix is fixed, or at least decreases sublinearly in the number of people he threatens to kill. He squares the number again and again, until the potential loss of lives weighted by the low probability that he's telling the truth actually outweighs a loss of $5. For good measure, he multiplies this number by a trillion. If you want to stick to your decision theory beliefs, you have to give him the money. Note that this argument does not rely on a benevolent Pascal: the mugger could just as easily threaten to torture Pascal himself for $n$ days, for some sufficiently large $n$.

Intuition says that if somebody threatens you with a Pascal's Mugging, you should laugh at them. That is, the preference relation that most of us hold says that you shouldn't listen to the mugger. Expected value says you should give him the money. There are two possibilities here: our preference here is irrational, perhaps as a result of intuition breaking down when numbers get too big; or the von Neumann-Morgenstern theorem does not correspond to the set of rational preference functions.

Here's another example: you're given a chance to flip a coin. If it lands heads, the consequence will be a world with 2.1x as much utility for you as there is right now. If it lands tails, the consequence will be a world with 0 utility for you. You can flip the coin as many times as you want, including 0. Naturally, the game ends once you land tails, because at utility 0 it doesn't matter what you multiply it by anymore. The expected value of one flip is \[ \mathbb{E}(p(x)) = (2.1x \cdot 0.5) + (0x \cdot 0.5) = 1.05x, \] where $x$ is your current utility. And the situation is the same on every flip! The expected value of flipping $n$ times is $1.05^nx$, which approaches infinity as $n$ does. Thus, the expected value maximizer flips the coin over and over until it comes up tails. Despite the expectation of infinite utility, it is easy to see that the probability that such an agent ends up with 0 utility, i.e. that the coin eventually lands tails, is 1. Intuition (at least mine) says that you should not take an action which has probability 1 of an outcome with 0 utility. As before, either our intuitive preferences are irrational, or von Neumann-Morgenstern rules out some preference relations that are in fact rational.

I claim that the preference to not give money to Pascal's Mugger, and to not flip the coin infinitely many times, are rational, and in fact the reverse preferences are irrational. Recall the very definition of a preference relation: a relation that is connected and transitive. The connectedness condition simply ensures that the preference relation is useful, i.e. that all pairs of items can be compared, but the transitivity condition is meant to capture a notion of rationality. An agent with an intransitive preference relation can be turned into a money pump: they have some preferences $x \le y, y \le z, z < x $, so if they start with $x$, you can freely trade them for $y$, then for $z$, then sell them back their $x$ for $z,$ over and over again. This understanding that a money-pumpable agent is irrational is actually baked into the von Neumann-Morgenstern framework -- it works with preference relations defined this way.

But Pascal the expected utility maximizer is also money-pumpable! You can ask him for five bucks -- or really, all the money he has -- then think of a number $n$ big enough that the threat of creating and killing $n$ Matrix-people, or of torturing Pascal for $n$ days, or whatever, that the higher-EV move is to give you the money. Similarly but less directly, the maximizer who flips the coin as many times as they can is giving up their $x>0$ utility for 0 utility. The clever reader may see how we could turn this into a money-pumping situation.

If it is irrational to be a money pump, and von Neumann-Morgenstern expected value maximization demands that you prefer to be a money pump in certain situations, then the conditions this theorem imposes on preference relations must fail to be conditions of rationality. They both include certain irrational preference relations, e.g. one that includes the preference to get Pascal's Mugged, and exclude certain rational preference relations, e.g. one that prefers to not have that happen.

In particular, I claim that the Archimedean Axiom is the one that errs: \[ p < q < r \implies \exists_{a,b\in(0,1)} \text{ s.t. } ap+(1-a)r < q < bp+(1-b)r. \] The proof that the Archimedean Axiom fails for an agent who doesn't give money to the mugger is left as an exercise. To lend some intuition, notice that the probability function for not giving him the money, $p_1,$ has $\mathbb{E}(p_1) = -\epsilon T$, where $-T$ is the utility of the immense threat he makes and $\epsilon$ is the probability that he's telling the truth; while the probability function for giving him the money, $p_2,$ has $\mathbb{E}(p_2) = -c$ for some small constant $c$ representing the utility of five dollars. If we don't give him the money no matter how big his threat is, i.e. for any $T$, we must have that $\epsilon T \le c$.

✦

We have seen that the decision-theoretic notion of utility is questionable: a property supposedly meant to identify better and worse outcomes, it turns out that maximizing it in expectation appears to be irrational, if we take money pumping as a sufficient condition for irrationality. This raises interesting questions about the notion of utility in general. A rational, self-interested agent would prefer not to maximize his own expected utility in the situations we have seen -- are these just "edge cases" somehow, qualitatively different from the day-to-day EV maximization problems we face? Or do these problems expose an underlying problem with the strategy of maximizing expected utility in general? Perhaps EV maximization just so happens to usually tell us the right thing to do, but for an entirely wrong reason. And beyond self-interested decision making, entire ethical systems are built around the concept of utility. How can an ethical theory posit that utility is the fundamental object of moral concern, the ultimate maximand, if it is not the case that a rational agent always wants maximum expected utility?

With respect to the von Neumann-Morgenstern framework, utility is a pretty simple concept -- it's a scalar-valued function defined simply as the value function $v(\cdot)$ applied to the sharp probability function $p_z$ of an outcome $z$. The discussion about the failure of maximization hints that it's actually a little more complicated than meets the eye, even in the formal setting; but beyond formal decision theory, it's even more unclear what sort of a thing it is. Can a person's utility be increased arbitrarily, or are there limits on how much utility a person can experience? Perhaps our ability to feel is capped proportional to our neuron count times the strength with which they fire -- but maybe this cap is so high that utility is effectively unbounded. What is the utility of nonexistence? If utility is defined on the set of outcomes and it corresponds to a preference relation, surely it's defined on nonexistence: nonexistence is certainly an outcome, and we certainly have preferences regarding actions that might bring about nonexistence. But intuitions about utility tend to be evasive on the topic. We fear death in a way that we don't fear never having been born. We see it is obligatory to not kill and as noble to save lives, but the way we view choices regarding the creation of life is nowhere near analogous -- you're not committing a moral crime by wearing a condom. Is this just a misalignment of the moral intuitions we've evolved? If $x$ people exist today and human existence is a net good, would we double the utility of the world by making it so that $2x$ people exist? Is the most efficient way to do good to maximize population size (within environmental constraints)? Would it be better if there were 100 times more people in existence but everyone was much less well-off? Is there some objective 0 on the utility scale, such that positive utilities are good, negative utilities bad, and the scale is unique up to positive linear transformation? Or is there no such thing as good and bad but only better and worse, with a utility function unique only up to positive affine transformation?

Rather than helping us draw confident conclusions about the nature of utility, I think the problem of expected utility maximization in decision theory mainly serves to shed light on the notion of utility by making it clear that we really don't know what we're talking about. Without coming up with some sort of clever qualification on the nature of the utility function, it seems like a utilitarian would need to be willing to risk human extinction, if only a cruel superintelligence came along and offered him the chance to flip a coin to either create a second, equally happy world with 2.1 times the population of Earth, or to kill all humans immediately. Further, this utilitarian would have to be willing to drive the probability of extinction to 1 if offered this gamble over and over.

The pragmatist argues that when we rule out weird cases like Pascal's Mugging, maximizing expected utility is still the way to go. Sure, at the very least it is a useful heuristic. But it's unclear what the fundamental difference is between our absurd edge cases and normal situations, what reason we could use to give ourselves permission to ignore the edge cases. Perhaps there exists a principled way to draw the line. But perhaps utility is just a nice piece of fantasy, an idea constructed such that it tends to align with our intuitions about what should and shouldn't be done, but which does not meaningfully represent a real property of the world.

✦