(This post draws heavily on earlier writing co-authored with Jesse Clifton, but he’s not listed as an author since he hasn’t reviewed this version in detail.)

Should we always be able to say whether one outcome is more likely, less likely, or exactly as likely as another? Or should we sometimes suspend judgment and say “none of the above”, that the answer is indeterminate?

Indeterminate beliefs (often modeled with imprecise probabilities)^[1] could have far-reaching implications for anyone who cares about the distant consequences of their actions. Most notably, we might be clueless about how our decisions affect the long-term future, if our estimates of our net effects on long-term welfare ought to be severely indeterminate. Perhaps we don’t have reason to consider most interventions good in expectation for the far future, even if we also don’t have reason to consider them bad or precisely neutral in expectation.

Before we can assess the case for cluelessness concretely, then, we should see if rationality requires us to have (or “act as if” we have) determinate beliefs. Here, I’ll argue that the positive arguments for having determinate beliefs in general are uncompelling, and indeterminate beliefs motivate different decision-making procedures than determinate beliefs. That is, there’s a viable alternative to “going with your best guess”. By itself, this claim doesn’t imply large changes in cause prioritization. But in my experience, accepting that indeterminate beliefs are plausible and decision-relevant goes a long way in making the case for cluelessness compelling.

Key takeaways:

The “degrees of belief” studied in this post are not, e.g., our acceptable betting odds, or a probability distribution that (along with a utility function) rationalizes our preferences. Rather, they are our basic judgments of the plausibility of different possible outcomes. And the rationality of our decisions under uncertainty depends on these beliefs. (more)
A prima facie motivation for having indeterminate beliefs in some situations is that a determinate judgment would be too arbitrary: Given lots of conflicting considerations, we might feel that the available information doesn’t give us reason to pin down one way of weighing up these considerations vs. some other ways. (more)
Indeterminate beliefs shouldn’t be confused with other ways of rejecting naïve Bayesianism, such as adjusting for model uncertainty, distrusting exact numbers, or modeling the instability/non-resilience of our credences upon reflection. None of these capture the key hallmark of indeterminacy, that is, suspension of judgment. (more)
The most straightforward extension of Bayesianism to indeterminate beliefs represents our beliefs with a set of probability distributions. And according to a natural extension of expected utility maximization to this case, the “maximality” rule, we prefer action over $B$ if and only if $A$ has higher expected utility under every distribution in the set. (more; more)
- But our beliefs are, arguably, fundamentally more vague than this. So any formalization of indeterminate beliefs will require some arbitrary choices, like specifying the exact boundaries of the set of distributions. Yet this formalization still involves less arbitrariness than precise probabilities. (more)
- And, when choosing between options that aren’t ruled out by maximality, we can choose based on normative considerations other than aggregate utility, such as non-consequentialist constraints. This is a crucial way in which indeterminate beliefs are decision-relevant. (more)
If you’d prefer to go with a given estimate as your “best guess” when forced to give a determinate answer, that doesn’t imply this estimate should be your actual belief. We don’t necessarily lose information by refusing to narrow down to a determinate estimate, including with a more sophisticated “meta-distribution”. (Indeed, often there isn’t a privileged way to do so.) Rather, we preserve the information that we lack sufficient reason to commit to a particular guess. (more)
The empirical track record of precise forecasts in domains like geopolitics leaves it an open question how precise our credences should be. The optimal degree of (im)precision is especially severely underdetermined when extrapolating to novel domains that pose qualitatively different epistemic challenges. (more)
Under severely indeterminate beliefs, the maximality rule permits a very wide variety of actions from an impartial consequentialist perspective. While this is counterintuitive, arguably the problem lies in the highly ambitious scope of impartial consequentialism itself, not with the decision rule. And even if a decision rule is counterintuitive, this doesn’t justify ad hoc changes to our beliefs. (more)

Background on degrees of belief and what makes them rational

(This section might sound pedantic, but it seems necessary to avoid talking past each other.)

We’d like to understand what it means to make good decisions under uncertainty. I’ll argue that, often, indeterminate degrees of belief (“indeterminate beliefs” for short) can be a rational way of characterizing this uncertainty. What kind of claim is this?

First, by “degrees of belief” or “credences” I mean “the weights we give to various possibilities based on their subjective plausibility, which we deliberately take to be inputs into our decisions”.^[2] This is in contrast to:

The credences of some ideal version of ourselves. Even assuming an unboundedly rational agent ought to have determinate credences, this doesn’t tell us what our credences should be, given the qualitatively different epistemic challenges we face;
Betting odds that we’d accept, which reflect hypothetical decisions that we’d make in response to our beliefs. Even if we only care about beliefs as far as they inform our decisions, we’re still interested in why we should make one decision vs. another, so we might want to reflect on epistemic norms (see footnote^[3] for more discussion);
A probability distribution with respect to which we can rationalize an agent’s behavior. For example, Savage’s theorem says that if your preferences satisfy certain axioms, you can be represented as maximizing expected utility under some probability distribution. But this distribution need not correspond to your beliefs;^[4]
The tools we use to communicate our epistemic state. For example, I’m not interested in answering, “If your all-things-considered belief is that AI doom is 40% likely, but you feel unconfident in this number, should you report a range [35%, 45%]?” I’m interested in whether the structure of your all-things-considered belief should be more like “40%” or “[35%, 45%]”, which I’ll unpack below.

Second, as argued in “Winning isn’t enough”, we can’t tell which beliefs or decisions are rational based purely on objective performance. Dutch books, complete class theorems, etc. only impose very weak constraints on our degrees of belief as defined above. And it’s question-begging to claim that certain beliefs “outperform” others, if we define performance as leading to behavior that maximizes expected utility under those beliefs. For example, it’s often claimed that we make “better decisions” with determinate beliefs.^[5] But on any way of making this claim precise (in context) that I’m aware of, “better decisions” presupposes determinate beliefs! Likewise, we can’t simply check which beliefs lead to higher utility on average over some distribution of toy problems, because this pushes the question back: Which distribution(s) represent the beliefs we should have about the real problems we face?

Rather, the case for/against determinate beliefs hinges on whether we endorse certain principles that aren’t about whether something “performs well”.

Motivating example

It’s out of scope here to argue that our beliefs about some particular practical questions should be (severely) indeterminate. That’s for another post. For now, here’s an example where we might want to refuse to compare the likelihood of two outcomes. I’m not confident that we should have action-relevantly indeterminate beliefs in this case or the examples listed below, and the concrete details are just for illustration. Thanks to Nicolas Macé for writing this vignette.

Say you want to tell whether working on pausing AI^[6] will decrease x-risk, or backfire. There are lots of relevant considerations. To name a few: A pause could have the intended effects of accelerating safety relative to capabilities. If alignment is difficult relative to capabilities, a pause could be necessary for alignment. Further, a pause could give our social and political institutions extra time to prepare for advanced AI. On the other hand, the pause could occur when frontier models are still too different from those posing a threat for the pause to translate into significant alignment progress. Combined with the fact that the pause might create compute and/or algorithmic overhang, it could turn out to be net-negative. Or, the pause could be more likely to be obeyed by safety-concerned actors, giving bad actors time to become more dangerous.
What do you do with all this? You try to hold in your head a foggy picture of how AI development might proceed, which actors involved in the process have which incentives, how different takeoff scenarios affect prospects for alignment, etc. Given that picture, you try to weigh up the plausibility of the hypotheses above, trying out different kinds of reference class forecasting and updating (or not) on different smart people’s views. After lots of thought, you don’t feel that the evidence and your priors (whatever those are) determinately support one side of the debate more than the other. But the cases for and against the pause also don’t seem equally plausible to you. You simply aren’t able to say whether pausing AI is more likely to decrease or increase x-risk. If someone forced you to give a best guess one way or the other, you suppose you’d say “decrease”. Yet, this feels so arbitrary that you can’t help but wonder whether you really need to give a best guess at all…

Some other illustrative examples of sign uncertainty about interventions aimed at reducing x-risk, where it might be appealing to suspend judgment (h/t Jesse Clifton):

Davidson: Human takeover could be worse than AI takeover.
Soares: Even if some research that advances AI capabilities also more strongly advances alignment, this research will increase x-risk if alignment is bottlenecked by serial research time.
Dai: AI control could accelerate the AI race more than it assists alignment research.
Trammell: Even if the future is good in expectation, plausibly the future is bad conditional on us having counterfactually prevented a given x-risk.
Steiner: Public benchmarks for ML capabilities may help AI companies improve models with a high-quality training signal, more than they help with AI safety policy.

The structure of indeterminacy

If you say the likelihood of $X$ compared to $Y$ is indeterminate, this means your answer to “Is $X$ more, less, or equally likely as $Y$ ?” is “None of the above”. (Whereas if you have determinate beliefs, you need to give one of those three answers.) Before we look at whether this kind of judgment could be rational, let’s get clear on what indeterminate beliefs look like, compared to other epistemic attitudes.

In the example above, our hapless protagonist’s reaction to a confusing pile of information was to suspend judgment, rather than give a determinate answer. You might wonder, as Soares suggests,^[7] aren’t there sophisticated classical Bayesian ways to make sense of this reaction?

But “it’s indeterminate whether $X$ is more likely than $Y$ ” goes beyond, or contradicts, various other responses to the limitations of (naïve forms of) classical Bayesianism:

I’m not simply recommending wider confidence intervals, or more determinate adjustments for model uncertainty, or more research and deliberation to improve non-resilient estimates.
I’m not merely saying that agents shouldn’t have precise credences when modeling environments more complex than themselves (cf. infra-Bayesianism).
I’m not saying that we would have a determinate credence after thinking more, and indeterminate credences are just credible intervals for that credence.
I’m not saying that there is some ideal credence we should have, but it’s unwise for humans to trust exact estimates of it. Bounded agents can’t always furnish precise estimates, and even when they can, needn’t have their actions dictated by those numbers. This is, of course, acknowledged by precise Bayesians.
I’m not endorsing any of the alternatives to Bayesianism that Ngo discusses here.
And, I’m certainly not endorsing “vibes-based analysis”. I’ll argue that instead of either pinning down a determinate credence, or following our whims and fuzzy common sense, there are other options.

Each of these responses either remains consistent with having determinate beliefs in (nearly) all cases, or abandons the project of rational belief formation entirely. They don’t capture the idea of suspending judgment per se. So, what’s the alternative?

Indeterminate Bayesianism

Having indeterminate beliefs doesn’t mean giving up on measuring our degrees of uncertainty. We don’t need to say we have Knightian uncertainty about everything we refuse to assign a single probability to. As Alexander puts it, “imperfect information still beats zero information”.

Enter imprecise probabilities. Suppose we want all the nice properties of Bayesianism as usual, but we allow the likelihood of one hypothesis compared to another to be indeterminate. Then we can represent our beliefs with a set $P$ of probability distributions called a “representor”.^[8] And we’ll say that $X$ is at least as likely as $Y$ if and only if $p (X) \geq p (Y)$ for every $p$ in $P$ . (See appendix for more formal details.) Given this representor, we write our credence in a given hypothesis as an interval. In the example above, let’s say your representor is a set $P$ of distributions over possible worlds such that your credence for “pausing AI would on net decrease x-risk” is $[0.45, 0.65] .$ It’s tempting to put a meta-distribution on the representor to get determinate credences anyway, but I’ll discuss below why this is unsatisfying.

A representor still seems to be an imperfect representation of our epistemic state, though. Humans don’t have anything close to a set of distributions over precisely specified possible worlds in our heads. And, if our worry about pinning down our credence in some hypothesis to a single number is that this is arbitrary, does it really help to pick two apparently arbitrary numbers rather than one? Instead of $[0.45, 0.65]$ , why not $[0.42, 0.66]$ , or $[0.5, 0.7]$ ?

I don’t know what the ideal model of indeterminate beliefs is. For now, we can take the representor as a starting point that’s less bad than a precise probability distribution at describing the qualitative constraints our beliefs satisfy. For example, if you write your credence in $X =$ “pausing AI decreases x-risk” as $[0.45, 0.65]$ , this tells me that (1) you think the likelihood of $X$ compared to not- $X$ is indeterminate, and (2) it would take stronger evidence to update you to “ $X$ is less likely than not- $X$ ” than to “ $X$ is more likely than not- $X$ ” (since the upper bound is further from 0.5 than the lower bound).

The exact boundaries of these constraints, though — how much stronger evidence exactly? — might be vague. To say that a category is “vague” means there are cases that elude classification as part of that category or not. (The classic example is the Sorites paradox.) In the example above, we may not want to decisively classify 0.45, 0.42, or 0.5 as inside or outside the interval. So, the vagueness of our credences explains why interval-valued representations remain unsatisfying, even though their ability to capture indeterminacy is an improvement over precise credences. Whichever way vagueness might be modeled, I tend to think that vague epistemic norms, by their nature, do not admit any perfectly accurate formal representation.^[9]

Why use the representor model, then? Because in order to make decisions, we often need some non-vague representation of our beliefs. Sometimes all we need to make a decision is a very coarse description of our beliefs. But in more complex contexts, it would be nice to have a formal representation of our beliefs that we can plug into a decision procedure. If our epistemic principles really are vague, the best we can do is to formally specify beliefs that exhibit the qualitative (vague!) virtues called for by our principles. The arbitrary endpoints of an interval-valued credence aren’t ideal. But imprecision can at least reduce the arbitrariness, compared to precise credences. Having to make some arbitrary choices when formally specifying our beliefs doesn’t mean we might as well make all of our choices arbitrarily!

Decision-making: A first pass

How do we make decisions when we have indeterminate beliefs? If we can’t always say whether $X$ is more likely than $Y$ , we can’t just maximize EV. What if one distribution in our representor says an intervention is more likely to result in the very good outcome $X$ than the (equally) very bad outcome $Y$ , but another distribution says the opposite?

The most natural starting point is the maximality rule. Maximality says that $A$ should be strictly preferred to $B$ if and only if $A$ has higher EV than $B$ under every distribution in your representor. Surely you should, at least, not do something that’s bad by the lights of every precise distribution in your representor! But “only if” might seem too permissive. Why not, say, aggregate the representor into one distribution and maximize EV as usual? I’ll address this concern below (here and here), and say more about narrowing down choices beyond the constraints of maximality in the next section.

Practical hallmarks of indeterminacy

We now have a basic framework for indeterminate Bayesian beliefs and decision-making. As noted above, the justification for these beliefs is fundamentally epistemic, not pragmatic. Still, we ultimately only care about the structure of our beliefs because they guide our decisions. Indeterminate beliefs have decision-relevant implications that qualitatively differ from those of typical EA/rationalist epistemic norms. I’ll focus on two of these here. (Even if you can model the following kinds of behaviors as consistent with precise Bayesianism after the fact, this doesn’t tell you what to do ex ante; see this post.)

Insensitivity to mild sweetening

Back to the example of pausing AI. Suppose that, as one notable input into your overall beliefs, you defer a bit to the nearly 34,000 signatories of the FLI open letter supporting the pause. Nonetheless, you think the sign of pausing AI is indeterminate. But let’s say you wait a while, and the letter now has 40,000 signatures. All else equal, this means you have stronger evidence in favor of pausing AI based on deference — the “pausing AI decreases x-risk on net” view has been “mildly sweetened”.^[10] Should you now favor that view, as if you’d previously believed pausing AI is equally likely to decrease x-risk as to increase it? Arguably not. Given your previous evidence, we suppose, you ought to have no clue which effect is more likely, and a bit of evidence this small plus no clue ought to equal no clue.^[11] This doesn’t mean you throw out the information from the mild sweetening, since you still update the distributions in your representor. It’s just that this update doesn’t suffice to make you think pausing AI determinately decreases x-risk.

Insensitivity to mild sweetening has implications for the value of information (VOI) of some kinds of research. Suppose you want to arrive at a determinate judgment on the sign of pausing AI. Perhaps you think, “It’s intractable to reduce my deep uncertainty about some features of AI takeoff, but not all of them. So I’ll research the tractable parts.” But, assuming the sign of pausing AI is still sensitive to the other features, you’d still consider the sign indeterminate after researching the tractable features. Hence this kind of research, and various other AI safety strategies that assume more information always strictly improves our decisions, would be futile.

Still, isn’t there some chance that if we research a topic that seems relevant to improving the far future, we’ll be better able to have a very large and positive impact? It’s out of scope for this post to dig into this more concretely. Briefly, however, at some point the upsides of research become so unlikely that they can easily be outweighed by seemingly speculative downsides. Maybe, by sharing your findings, you inspire some readers/listeners to push for worse versions of a pause than the default, or you displace funding for more effective interventions, or you antagonize people at AI labs who would’ve otherwise supported alignment efforts.

More generally, if maximality permits both $A$ and $B$ because $A$ is better under some distributions while $B$ is better under others, you’re not indifferent between them. So a new consideration in favor of $A$ having higher expected total utility doesn’t imply you should do $A$ .

Suspending judgment on total effects, and choosing based on other reasons

You might think that it doesn’t matter if you have indeterminate beliefs, or consider multiple actions permissible under maximality, because you have to make a choice anyway. Doesn’t this choice reveal your implicit determinate credence/preference? E.g., for any proposition $X$ that your choice depends on, it seems you can always ask yourself, “How much would I bet on $X$ ?”. From Soares:

But from another perspective, every decision in life involves a “bet” of sorts on which action to take. The best available action may involve keeping your options open, delaying decisions, and gathering more information. But even those choices are still “part of the bet”. At the end of the day, you still have to choose an action.
Humans can’t generate precise credences. Even our fuzzy intuitions vary with framing and context. Our reasoning involves heuristics with known failure modes. We are subject to innumerable biases, and we can’t trust introspection. But when it comes time to act, we still have to cash out our uncertainty.

But we should distinguish our beliefs about our options’ total consequences from our beliefs about what’s best to do overall. For example, even if you’re highly sympathetic to impartial consequentialism, you might make decisions some way other than by purely maximizing expected total welfare. If so, then in order to make a choice, you don’t need to determinately rank options by expected total welfare.

Instead, you can narrow down options based on their total effects only as far as you have reason to, and no further. (I’ll say more on why not to narrow down to your supposed “best guess” below; here I’m just focusing on why indeterminacy matters for decisions.) That is, if $A$ and $B$ are both permitted by maximality, you suspend judgment on whether $A$ is better for total welfare than $B$ (cf. Bradley (2012, sec. 5.7)). From there, you can choose based on other reasons, instead of picking arbitrarily. This suggests a response to the problem of consequentialist cluelessness other than “make up some numbers and maximize EV”.

What would these other reasons for choice be? This is partly an open question. But recall the reason why you’d have indeterminate beliefs in the first place: your intuitions and evidence give you reason to narrow down your estimates of the total effects only so far. So, compared to effectively picking an estimate from a hat beyond that, presumably various other reasons for choice could easily be better. I’ll offer two proposals, one object-level and the other meta.

Moral uncertainty / pluralism. Assume we’re stuck with maximality as far as impartial consequentialism goes. We could break ties between the remaining options based on moral views that aren’t impartially consequentialist.^[12] It’s unclear what these views recommend under indeterminacy about the long term, insofar as they still care about consequences. We should avoid quick appeals to common sense here. But we might find fairly robust takeaways. For example, a variety of views say we have a duty to address ongoing moral atrocities, which pushes towards working to end factory farming. And if we don’t have reason to think we can have very large expected positive impact, we should give more weight to deontological side constraints. On the other hand, what would justify longtermist strategies other than our beliefs about their long-term consequences? Presumably “duties to address moral atrocities” don’t tell us to take action if we could just as well make the future atrocity worse, given our limited knowledge — which is our predicament if our beliefs about the far future are severely indeterminate.

Reflection on our epistemic situation and decision-making. Rejecting determinate beliefs is a fundamental shift in perspective on our altruistic prioritization. We don’t immediately know the implications of our empirical evidence and normative intuitions for what kinds of beliefs we should have, or what kinds of decisions we should make. So shouldn’t we work out these implications enough to get our bearings? In the context of reasoning about Bayesian model uncertainty, Violet Hour writes:

[Imagine] that I’m at the poker table, but this time I don’t know the rules of poker, nor do I know the composition of a standard deck of cards, or even the concept of a ‘card game’. Here’s one thing I could do: start by trying to construct explicit credences over my current, highly confused model of the state space, and choose the highest expected value action, based on my current understanding of what’s going on.
I wouldn’t do this, and probably nor would you. In practice, I’d be much more tempted to focus my efforts on first constructing the relevant state space, over which I may then start to assign more explicit probabilities. I’d ask various questions about what we’re even doing, and what the rules of the game are.

Similarly, it seems reasonable to say “when we’re confused as to what it even means to make good decisions, deliberate more on this”, even if we can’t cash this out in precise VOI terms. For example:

Are there any interventions whose effects on total welfare are net-positive even under the kinds of indeterminate beliefs we should have?
If maximality gives very little action guidance, what about the minimax regret rule, or some kind of hedging?
Is there a way of making sense of the intuition that we should focus on the most “robust”, foreseeable consequences of our decisions?

Whether these particular research questions are promising in absolute terms is beside the point. They are relatively promising compared to arbitrary guesses, because they at least attempt to take our epistemic challenges seriously.

Responses to key objections to indeterminacy

As argued here and summarized here, you can have indeterminate beliefs and still avoid dominated strategies, without acting contrary to your preferences.^[13] Here are other common arguments for determinate beliefs, and why I think they don’t work. This isn’t comprehensive, and I plan to address other objections in a future post.^[14] But my impression is that other objections are secondary for this audience. To reiterate, this post isn’t meant to argue that we should have significantly indeterminate beliefs about the far future in particular.

“Aggregating our representor with higher-order credences uses more information”

Suppose we feel inclined to represent our beliefs with a set of distributions. Perhaps if we do this, we’ll lose information about the higher-order weights we’d endorse putting on these different distributions. So aggregating the representor with these higher-order weights might be better. Specifically:

Higher-order weights from intuition: Don’t we always have some intuitions about the relative plausibility of different distributions in our representor? (For example, if you initially want to represent your credence in “pausing AI reduces x-risk” as $[0.45, 0.65]$ , maybe with more thought you’d be more likely to converge to 0.45 than 0.65.) If so, shouldn’t we aggregate our representor with weights that reflect these intuitions?
Equal higher-order weights: As recommended by the principle of indifference, if we don’t know how to weigh up these distributions, shouldn’t we give them all equal weight and aggregate them into one distribution?

Responses

Insofar as our representor exhaustively describes our epistemic state, it’s not clear what these higher-order weights are supposed to mean.^[15] Each distribution in the representor models a different way of weighing up the relevant considerations, and, by hypothesis, we think it’s indeterminate which way is best. So why is the aggregated representor, derived from putting determinate weights on each distribution, a better description of what we ought to believe? By aggregating, instead of adding information for free, it seems we lose information about our inability to non-arbitrarily pin down weights.

Still, maybe the reasons in favor of the determinate higher-order weights are strong enough to outweigh this cost. Let’s look at the two approaches for setting higher-order weights above.

Higher-order weights from intuition. Often, the same kind of indeterminacy we saw in our first-order uncertainty will recur in our higher-order uncertainty. Different intuitions will push in different directions, and you may not be able to determinately weigh them up.

But let’s grant that the answer you’d give with a “gun to your head” is privileged over other determinate answers. Why is it privileged over an indeterminate answer? You can update on the evidence “I currently intuit that I’d endorse [these higher-order weights] more than other possible weights”, without making your updated credence itself precise. The fact that you should go with your intuition when forced to give a precise number doesn’t imply you should aggregate all things considered. (Recall that we aren’t forced to aggregate when making decisions, because when multiple options are permissible with respect to their expected total consequences, we can resolve indeterminacy with other normative considerations.)

Perhaps the claim is, your gun-to-your-head intuition is always at least slightly truth-tracking, and this is a strong reason to aggregate according to that intuition. It’s not exactly clear how to operationalize “truth-tracking” here. But in any case, due to systematic biases, our intuitions can point away from the truth in various domains:

Tetlock (2017) infamously found that “dart-throwing chimps” beat undergrads at forecasting.
People can be systematically overconfident in stock market predictions (Torngren and Montgomery 2004; Deaves, Lüders, and Schröder 2010).
Our intuitions about features of the far future could simply be biased by our lack of access to these features, since this makes it less plausible our intuitions track far future mechanisms.

To be clear, this isn’t to say our intuitions are overall worse than chance. The point is, if there are reasons our intuitions about an unfamiliar domain could systematically deviate from the truth, and we don’t know how to weigh these up against the reasons for truth-tracking, it seems indeterminate whether our intuitions are truth-tracking overall!

Equal higher-order weights. As is well-known, the principle of indifference (POI) often gives different answers depending on which of several apparently arbitrary re-framings of the problem you use. This means that often there’s no privileged way to assign equal weights.

Take, for example, our interval $[0.45, 0.65]$ for P(“pausing AI would decrease x-risk”). One way to apply the POI is to take the midpoint, 0.55. But where did that interval come from? Implicitly, we formed some model in our head of pathways from “AI pause” to “misaligned AI takeover is prevented”. Then we plugged a range of estimates of some parameters into the model, to get a set of probabilities. So here’s another way to apply the POI: Take the midpoint of the range of each parameter first, and plug those into the model. Or, why not apply the POI to a different model that has a different breakdown of “parameters”? (See appendix for more.)

Even if we can find a non-arbitrary way to assign equal higher-order weights,^[16] what’s the positive motivation for doing this instead of suspending judgment? Contrast this with using the POI to set first-order credences in, say, the result of a fair coin flip: There’s an unambiguous symmetry in the possible outcomes, heads vs. tails, so our credence in each outcome should be the same. Different methods of assigning first-order weights don’t seem symmetric in this way — again, by hypothesis, we have reasons in different directions favoring different weightings, which aren’t equally good.

Finally, even if there’s a privileged way to aggregate a precisely specified representor, our beliefs will often be too vague for this to help us. As discussed above, we might find it arbitrary to commit to the interval $[0.45, 0.65]$ for our credence in “pausing AI would decrease x-risk” instead of, say, $[0.42, 0.66]$ or $[0.5, 0.7] .$ But then, the midpoint will be vague as well.

“Precise forecasts do better than chance”

Empirically, the Brier scores of precise forecasts are better than chance. Indeed, superforecasters can do significantly better than chance in geopolitical forecasting, which you’d think faces enough murkiness to warrant imprecision. Against using imprecise probabilities in forecasting, Lewis writes:

Empirically, with forecasting, people are not clueless. When they respond to pervasive uncertainty with precision, their crisp estimates are better than chance… In the same way our track record of better-than-chance performance warrants us to believe our guesses on hard geopolitical forecasts, it also warrants us to believe a similar cognitive process will give ‘better than nothing’ guesses on which actions tend to be better than others, as the challenges are similar between both.

Responses

I’m not sure exactly what Lewis’s argument is meant to be. On the most straightforward interpretation, this is a non sequitur, since indeterminate forecasts aren’t the same as randomly chosen determinate forecasts. If you ask someone who endorses an imprecise credence, “Which number in the interval would you go with, if you had to pick one?”, maybe they’d pick at random (though not necessarily, as noted above). But that’s not the same as asking what their forecast is.

A more charitable reading is: The fact “superforecasters’ precise credences significantly beat chance in a challenging domain” is a reason to think that if superforecasters had severely imprecise credences in this domain, they’d fail to capture information.

Ideally, we’d want to dig into what exactly “capturing information” means. For the purposes of this post, though, I’m happy to grant the spirit of this claim, and give some general responses that I think should apply to any reasonable operationalization of the claim. In short, the claim doesn’t imply that our credences about our impact on the far future should have a high degree of precision, or that our credences shouldn’t have any imprecision.

First, given our calibration (if any!) in domains like geopolitical forecasting, what can we conclude about the kinds of beliefs we should have about domains we care about as effective altruists? To answer this, we need to form beliefs about how much the former transfers to the latter. And arguably these beliefs should themselves be highly indeterminate, given that we have no direct evidence that superforecasters are calibrated about the far future.^[17] I won’t get into detailed reasons to have severely indeterminate beliefs about the far future, until a future post. But there are some prima facie significant differences between geopolitical forecasting and the kinds of forecasts EA decision-making depends on. For example, for the latter, we need to model extremely complex and unprecedented causal pathways, like the effects of our actions on a post-ASI civilization. And our decisions are sensitive to small differences in probabilities, as well as subtle philosophical crucial considerations.

Second, the reason we care about the question of indeterminacy is that we want to make good decisions. These decisions depend on our beliefs about many variables at a time. So, even if our credence in each individual variable is only moderately indeterminate, we can still be left with severe indeterminacy in the expected values of candidate actions.

Lastly, this isn’t an argument against having indeterminate credences at all in the given domain. I’m not aware of evidence that superforecasters benefit from arbitrarily many decimal places of precision. If we have evidence that additional precision on the order of, say, two decimal places is informative, then to say more, we once again need to form beliefs about how to extrapolate from this evidence. And, on its face, this matters for forecasting rare events in the far future, where “noise” on the order of three or more decimal places can easily drown out the signal.

“Maximality is too permissive”

Recall that the most straightforward generalization of EV maximization to indeterminate beliefs, maximality, only prohibits actions that are worse in expectation than some alternative according to every probability distribution in your representor. We saw above that in many cases, it doesn’t make epistemic sense to try to aggregate the representor into one distribution. Given this, it also seems unmotivated to simply maximize EV with respect to a distribution given by aggregating the representor.

So if we have severely indeterminate beliefs about the long-term consequences of our actions, then insofar as we care (roughly) impartially about those consequences, maximality will give very little action guidance. Indeed, Mogensen (2020) argues that maximality leaves many of our altruistic choices indeterminate. He compares the Against Malaria Foundation with the Make-a-Wish Foundation, pointing out that distributing bednets might lead to an increase in the rate of population growth, which would have many extremely difficult-to-weigh-up effects. In addition to there being more people, which directly affects net welfare, population growth could strain natural resource availability, reduce wild animal populations, lead to more technological innovation, and accelerate economic growth. Each of these has myriad flow-through effects, possibly including effects on the long-run future of civilization (e.g., via effects on extinction risk), which probably dominate the calculus on any given precise belief state. It is very likely that there are plausible ways of weighing up these considerations — i.e., precise beliefs that are strong candidates for inclusion in a representor — that say Against Malaria is worse than Make-a-Wish, and some that say it’s better, so maximality makes both permissible.

For some (e.g., Chappell and Thorstad and Mogensen (2020, sec. 4)), this implication is a reductio of indeterminate beliefs.

Responses

First, note that maximality doesn’t seem to give counterintuitive verdicts with respect to mundane, local goals like “don’t get terribly injured”. No plausible precise distribution you might have in your representor would tell you that wandering around blindly in the road has lower risk of terrible injuries than taking the sidewalk. This means that if we find that maximality gives surprisingly little action guidance with respect to a more ambitious utility function, arguably the utility function is to blame. From Daniel:

I think that the common sense justification for not wandering blindly into the road simply is that I currently have a preference against being hit by a car. I don’t think the intuition that it’d be crazy to wander blindly into the road is driven by any theory that appeals exclusively to long-term consequences on my well-being, nor do I think it needs such a philosophical fundament. … I think that a large part of the force of the “new problem of cluelessness” (i.e., instances where the defence that “indirect effects cancel out in expectation” doesn’t work) comes from the contingent fact that (according to most plausible axiologies) impartial goodness is freaking weird.

Second, I’m not sure how this is supposed to be an argument against severely indeterminate beliefs per se. I think it would be motivated reasoning to work backwards from our intuitive moral judgments to epistemic conclusions. Remember, even if some action isn’t ruled out on impartial consequentialist grounds given our honest beliefs, we can still rule it out on other normative grounds. There may also be impartial consequentialist decision rules for indeterminate beliefs that are more action-guiding than maximality.

Conclusion

I hope to have illustrated why it isn’t always preferable to compress your uncertainty into a determinate “best guess”, even if you acknowledge the non-resilience of this guess. You can suspend judgment on some matters, without simply picking at random or losing information. And suspending judgment means something qualitatively different from merely thinking more before taking action, because you might expect your beliefs about some questions to be persistently indeterminate. But why, concretely, should our beliefs about the long-term future be so severely indeterminate that we’re clueless about a wide swath of altruistic strategies? I’ll consider this in a future post.

Acknowledgments

Thanks to Maxime Riché, Tristan Cook, Michael St. Jules, Jamie Harris, and Sylvester Kollin for feedback; Nicolas Macé for many helpful suggestions; and Jesse Clifton for various writings this post draws on. This does not imply their endorsement of all my claims.

Appendix: Indeterminacy for ideal agents

(This appendix was almost entirely authored by Jesse Clifton; I’ll use “we”, accordingly.)

When we talk about “ideal agents”, we’ll mostly be talking about logically omniscient agents. So, agents that are capable of instantaneously computing the expected utilities of different policies given priors over ~all relevant worlds to arbitrary precision.

As noted above, whether we should have indeterminate beliefs is a separate question from whether an ideal agent should. Still, the model of an ideal agent can be useful for thinking about the reasons why agents in general might (not) be justified in determinate beliefs.

So what kind of structure do beliefs have? Using Stefánsson’s (2017) terminology, mentalism holds that, for a rational agent, beliefs are fundamentally probability distributions, or sets thereof, in the agent’s head. Comparativism about belief instead holds that the fundamental structure of belief consists of judgments of the form “ $X$ is likelier / just as likely / less likely / [none of the above?] than $Y$ ”.^[18]

We don’t want to take a stance here on whether mentalism or comparativism is right. But we find it more natural to think about the foundations of (in)determinate credences in terms of the properties of comparative confidence judgments. In particular, we find it enlightening to think about the difference between determinacy and indeterminacy in terms of the completeness axiom, which is what distinguishes determinacy and indeterminacy in the comparativist framework (more on this soon). Comparativism also seems to apply more naturally to bounded agents, since comparative confidence judgments are easier than numerical judgments. So we’ll approach things from a comparativist point of view.

The question of what beliefs are rational is, then, a question of what properties these comparative judgments ought to have. A few natural-looking properties for comparative beliefs to have are (all of the capital letters are events, like “it rains tomorrow”):

Transitivity: If A is more likely than B and B is more likely than C, then A is more likely than C;
Completeness: For every A and B, A is either more, less, or exactly as likely as B;
Boundedness: If A is impossible, then every B is at least as likely as A. And if A is necessary, every B is at most as likely as A;
$\lor$ -Separability: If B and C are inconsistent with A, then: “A or B” is likelier than “A or C” if and only if B is likelier than C.

It turns out that comparative beliefs satisfying these properties, plus a “continuity” condition, can be represented by a (unique) probability distribution.^[19] That is, there is a probability distribution $p$ such that A is at least as likely as B if and only if $p (A) \geq p (B)$ . And so, from a comparativist point of view, the question of whether our beliefs should be representable by a probability distribution boils down to whether we ought to accept these axioms of comparative belief.

So what if we dropped the completeness axiom, allowing agents to judge some A’s as neither likelier, just as likely, or less likely than some B’s? We would get an imprecise probabilist. Bradley (2017, chap. 11) shows that when comparative beliefs satisfy the above assumptions minus completeness, and an additional coherence requirement,^[20] they can be represented by a set of probability distributions. That is, there’s a set $P$ of probability distributions (called a “representor”) such that A is at least as likely as B if and only if $p (A) \geq p (B)$ for each $p$ in $P$ .

Indeterminate priors

Let’s suppose that the ideal agent’s comparative beliefs ought to satisfy all of the axioms discussed so far, except possibly completeness. So we can think of our agent’s beliefs at any time as being represented by a set of probability distributions. (If they satisfy completeness, $P$ will contain a single distribution.) The question we want to explore here is, Are there compelling non-pragmatic principles that recommend a unique set of complete beliefs, given a body of evidence?

We also need to specify how our agent responds to evidence $E$ . Precise Bayesianism, of course, generally entails updating the prior using Bayes’ rule. It’s not immediately clear that indeterminate beliefs should be updated by simply applying Bayes’ rule to each element of the representor.^[21] But our discussion of incompleteness isn’t going to hinge much on the choice of update rule, so for simplicity we’ll assume that they are. I.e., an agent's beliefs in hypotheses $H$ given all of their evidence $E$ are represented by the posterior $P (H | E) = {\frac{p (H, E)}{\sum_{H^{'}} p (H^{'}, E)} | p \in P}$ , where $P$ is their prior. With this update rule in place, we can see that for the posterior to be indeterminate, we need there to be indeterminacy in the joint prior beliefs in $H$ and $E$ .

For now, we’ll focus on the question: What kinds of principles should guide an ideal agent’s prior beliefs about hypotheses $H$ ? Do they pin down a unique prior (hence, imply completeness)?

(Note: The approach below might be either unfamiliar or profoundly distasteful to readers used to thinking in terms of Solomonoff induction when it comes to setting priors for ideal agents. Bear with us, we’ll get to Solomonoff soon.)

An ideal agent reasons about extremely specific hypotheses $H$ . For concreteness, we’ll consider hypotheses made up of the following pieces. We aren’t asserting that the ideal agent will cast hypotheses into this exact framework. The important part, as we’ll see, is that the ideal agent fully specifies the ontological content of each theory, and uses this to guide its prior-setting.

an ontology, which tells us the fundamental entities that make up the world and the properties they can have;
a spacetime structure (the mathematical structure of space and time — e.g., according to general relativity the spacetime structure is a particular kind of 4-dimensional Lorentzian manifold);
a set of dynamical laws telling us how the fundamental entities behave (deterministic, for simplicity);
some notion of “initial conditions” $x$ , a full description of the properties of the fundamental entities at a certain point in spacetime from which the conditions at other points in spacetime can be derived via $f$ .

Let’s take some ontology and spacetime structure as given, and think about principles for initial conditions $x$ and dynamical laws $f$ . The two most popular kinds of principle for specifying such beliefs are as follows.

The principle of indifference

The usual version of this principle tells us that we should set priors by spreading out our credences evenly over the space of possibilities. The motivation (we’d argue) is non-arbitrariness: If you don’t see any reason for an asymmetry among the possibilities, it would be arbitrary to give unequal weights to them.^[22] The principle of indifference is (implicitly) invoked in many arguments from “symmetry” (such as the idea of “simple cluelessness” (Greaves 2016)), induction, and anthropic reasoning (Bostrom 2003). For example, MacAskill’s critique of the “hinge of history” hypothesis argues that among a list of N centuries, our prior that each given century is the most influential should be 1/N.

Unfortunately, as you might have heard, attempting to get a unique answer with this principle faces serious difficulties when applied to continuous or unbounded spaces of possibilities. For unbounded spaces, there’s the simple problem that you can’t get a uniform probability distribution on an unbounded space. And then there’s the problem that applying the principle to apparently arbitrary re-descriptions of the problem will yield different beliefs. Take van Fraassen’s (1989) famous cube factory. You’re told that a cube factory makes cubes whose side length is between 0 and 1, and nothing else. What should your belief be about whether a cube from the factory has side length between 0 and ½? Surely ½, you might say, since by the principle of indifference our credence in side length should be uniform on [0, 1]. But wait. Couldn’t we make the same argument to get uniform credences over face areas between 0 and 1, yielding a credence of ¼ in a side length in [0, ½]? And why not the same for volume? Or any other power of side length?^[23] Arguably, you’re more justified in having a credence $(0, 1)$ in the proposition that the side length is between 0 and ½, than picking one specific number.

As a more realistic example, consider the hinge of history prior above. MacAskill picks a cutoff point in the far future for the last century, and takes a uniform prior over centuries. But as suggested by Ord, why not instead partition by all persons who will ever live, which is unbounded because we don’t know how many persons will exist in the future? And in that case, the uniform prior is undefined, so it seems our choice of prior for influentialness will be arbitrary.

A potential way out is to apply the principle of indifference at the most “explanatorily basic” level.^[24] It’s way out of scope to try to make that precise here, but the gist is that your theory comes with some notion of which entities, properties, processes, etc. explain everything else about the world. Explanation might involve causation — e.g., maybe you want to say that the initial conditions are explanatorily prior to everything that comes after, because they help cause the later events — but it might be other things too (metaphysical grounding??). So, the privileged domain to apply the principle of indifference to might be the possible values of the explanatorily basic variables, e.g., initial conditions of fundamental particles.

Even if we can get a determinate prior over initial conditions for every theory under consideration using the principle of indifference, there’s still the question of how to apply the principle to the case of dynamical laws. For example, how do we parameterize the family of laws of which Newtonian physics is a member? Maybe one parameter is the function of radius that appears in the gravitational force? What space of functions, and how do we put a uniform prior over it?

Occam’s razor

As usually formulated (at least outside the rationalist community), this principle tells us to prefer simpler ontologies, all else equal. For us, the most appealing motivation for Occam’s as an a priori principle is, again, its anti-arbitrariness spirit.^[25] If we think of it as arbitrary to entertain the existence of some metaphysical stuff we have no direct access to, we ought to commit as little arbitrariness as possible and so posit as little metaphysical stuff as possible, says Occam.

The tricky part comes when we have to be exact about how “simple” a theory is, and how to convert these measures of simplicity into beliefs about theories. Start with simplicity-as-number-of-fundamental-entities. Maybe counting fundamental entities is easy enough (the 17 fundamental particles of the Standard Model look like 17 fundamental entities to us), though of course this already requires metaphysical judgments, an ontology to go with the entity-less formalism of the theory. But how do we quantify, say, the complexity of a theory’s dynamical laws? What’s more complex, for any given $N$ : A physics with $N$ particles and 3 fundamental forces (call this theory $A_{N}$ ), or a physics with 17 particles and 4 fundamental forces (call this $B$ )? How much more or less complex?

This imprecision in our notions of “simplicity” look to us like a tempting place to apply, well, imprecision. It is very natural to imagine superintelligent versions of ourselves surveying the various theories on offer, being able to clearly judge some theories as simpler than others, but in other cases feeling that it’s indeterminate. I.e., an incomplete simplicity ordering, representable (hopefully, if other conditions are met) by a set of probability distributions.

(That said, recall our discussion of vague beliefs above. Even for an ideal agent, comparative beliefs about the simplicity of ontologies might be vague in this sense. It’s a bit odd to imagine that our agent thinks it’s definitely indeterminate whether $A_{26}$ is simpler than $B$ , but that $A_{25}$ is definitely simpler than $B$ . That might be kind of like saying a heap of sand must have at least 1,000,026 grains. It’s entirely plausible that the notion of “ontological simplicity” can (non-arbitrarily) be made precise enough that this works, but it’s also entirely plausible to us that it can’t be.)

Solomonoff induction. Solomonoff induction says, If you want a principled way of specifying an Occam-compatible prior over hypotheses, do this. First, we’re going to encode our hypotheses as algorithms that spit out sequences of observations, which are the data you’re seeing and updating on. (We’re generously setting aside the possibility of uncomputable hypotheses.) Then, we’re going to measure the complexity of each algorithm $a$ by its Kolmogorov complexity $K (a)$ . Finally, we’re going to specify our prior mass on $a$ as proportional to $2^{- K (a)}$ .

The problem is that Kolmogorov complexity depends on the language in which algorithms are described. Whatever you want to say about invariances with respect to the description language, this has the following unfortunate consequence for agents making decisions on the basis of finite amounts of data: For any finite sequence of observations, we can always find a silly-looking language in which the length of the shortest program outputting those observations is much lower than that in a natural-looking language (but which makes wildly different predictions of future data). For example, we can find a silly-looking language in which “the laws of physics have been as you think they are ‘til now, but tomorrow all emeralds will turn blue” is simpler than “all emeralds will stay green and the laws of physics will keep working”...

You might say, “Well we shouldn’t use those languages because they’re silly!” But what are the principles by which you decide a language is silly? We would suggest that you start with the actual metaphysical content of the theories under consideration, the claims they make about how the world is, rather than the mere syntax of a theory in some language. If you think there are strong independent reasons for using a Solomonoff prior, or set of Solomonoff priors, you could then try to fit a set of such priors to your judgments about comparative ontological simplicity.

But suppose, in the end, you don’t want to indulge in any of this metaphysical hocus pocus, and you want to go only on Solomonoff induction + your raw feelings about the reasonableness of different description languages. This still doesn’t get you to a determinate prior. Because you can have a raw feeling that many languages are natural, and indeed that would presumably be the default outcome, given how underdetermined these raw feelings of naturalness are. In that event it would seem reasonable to have an indeterminate prior consisting of many such priors, rather than effectively picking one (or some mixture) from a hat.

All in all, it seems well within the realm of possibility that normative principles we’d endorse on reflection would fail to single out one prior over the laws of nature.

References

Bostrom, Nick. 2003. “Are We Living in a Computer Simulation?” The Philosophical Quarterly (1950-) 53 (211): 243–55.

Bradley, Richard. 2017. Decision Theory with a Human Face. Cambridge University Press.

Bradley, Seamus. 2012. “Scientific Uncertainty and Decision Making.” London School of Economics.

———. 2022. “How to Be an Imprecise Impermissivist.”

Bradley, Seamus, and Katie Steele. 2014a. “Should Subjective Probabilities Be Sharp?” Episteme (Edinburgh) 11 (3): 277–89.

———. 2014b. “Uncertainty, Learning, and the ‘Problem’ of Dilation.” Erkenntnis. An International Journal of Analytic Philosophy 79 (6): 1287–1303.

———. 2016. “Can Free Evidence Be Bad? Value of Information for the Imprecise Probabilist.” Philosophy of Science 83 (1): 1–28.

Builes, David. 2022. “The Ineffability of Induction.” Philosophy and Phenomenological Research 104 (1): 129–49.

Climenhaga, Nevin. 2020. “The Structure of Epistemic Probabilities.” Philosophical Studies 177 (11): 3213–42.

Deaves, R., E. Lüders, and M. Schröder. 2010. “The Dynamics of Overconfidence: Evidence from Stock Market Forecasters.” Journal of Economic Behavior & Organization 75 (3): 402–12.

Easwaran, K. 2014. “Decision Theory without Representation Theorems.” Philosophers’ Imprint 14 (August). https://philpapers.org/rec/EASDTW .

Elkin, Lee. 2024. “Normative Uncertainty Meets Voting Theory.” Synthese 203 (6). https://doi.org/10.1007/s11229-024-04644-6.

Eriksson, Lina, and Alan Hájek. 2007. “What Are Degrees of Belief.” Studia Logica 86 (2): 185–215.

Fraassen, Bas C. van. 1989. Laws and Symmetry. Clarendon Press.

Greaves, Hilary. 2016. “XIV—Cluelessness.” Proceedings of the Aristotelian Society 116 (3): 311–39.

Gustafsson, Johan E. 2022. “Money-Pump Arguments.” In Elements in Decision Theory and Philosophy. Cambridge University Press.

Hájek, Alan. 2004. “Symmetry Is the Very Guide of Life.” 2004. https://fitelson.org/few/few_04/hajek.pdf .

Hájek, Alan, and Michael Smithson. 2012. “Rationality and Indeterminate Probabilities.” Synthese 187 (1): 33–48.

Hedden, Brian. 2015. “Time-Slice Rationality.” Mind; a Quarterly Review of Psychology and Philosophy 124 (494): 449–91.

Henderson, Leah. 2014. “Bayesianism and Inference to the Best Explanation.” The British Journal for the Philosophy of Science 65 (4): 687–715.

Huemer, Michael. 2009. “Explanationist Aid for the Theory of Inductive Logic.” The British Journal for the Philosophy of Science 60 (2): 345–75.

———. 2016. “Serious Theories and Skeptical Theories: Why You Are Probably Not a Brain in a Vat.” Philosophical Studies 173 (4): 1031–52.

Konek, Jason. 2019. “IP Scoring Rules: Foundations and Applications.” Edited by Jasper De Bock, Cassio P. de Campos, Gert de Cooman, Erik Quaeghebeur, and Gregory Wheeler. International Symposium on Imprecise Probabilities and Their Applications, Proceedings of Machine Learning Research, 103:256–64.

Lyon, Aidan. 2017. “Vague Credence.” Synthese 194 (10): 3931–54.

MacAskill, William. 2013. “The Infectiousness of Nihilism.” Ethics 123 (3): 508–20.

Meacham, Christopher J. G., and Jonathan Weisberg. 2011. “Representation Theorems and the Foundations of Decision Theory.” Australasian Journal of Philosophy 89 (4): 641–63.

Mogensen, Andreas L. 2020. “Maximal Cluelessness.” The Philosophical Quarterly 71 (1): 141–62.

Rabinowicz, Wlodek. 2020. “Between Sophistication and Resolution – Wise Choice.” In The Routledge Handbook of Practical Reason, 526–40. New York, NY, USA: Routledge.

Rasmussen, Carl, and Zoubin Ghahramani. 2000. “Occam’s Razor.” Advances in Neural Information Processing Systems 13. https://proceedings.neurips.cc/paper/2000/hash/0950ca92a4dcf426067cfd2246bb5ff3-Abstract.html .

Rinard, Susanna. 2017. “Imprecise Probability and Higher Order Vagueness.” Res Philosophica 94 (2): 257–73.

Schoenfield, Miriam. 2012. “Chilling Out on Epistemic Rationality: A Defense of Imprecise Credences.” Philosophical Studies 158 (2): 197–219.

Steele, Katie. 2007. “Distinguishing Indeterminate Belief from ‘risk-Averse’ Preferences.” Synthese 158 (2): 189–205.

Stefánsson, H. Orri. 2017. “What Is ‘Real’ in Probabilism?” Australasian Journal of Philosophy 95 (3): 573–87.

Tetlock, Philip E. 2017. Expert Political Judgment: How Good Is It? How Can We Know? - New Edition. Princeton University Press.

Thorstad, David, and Andreas Mogensen. 2020. “Heuristics for Clueless Agents: How to Get Away with Ignoring What Matters Most in Ordinary Decision-Making.” Global Priorities Institute Working Paper No. 2-2020, University of Oxford.

Torngren, Gustaf, and Henry Montgomery. 2004. “Worse than Chance? Performance and Confidence among Professionals and Laypeople in the Stock Market.” The Journal of Behavioral Finance 5 (3): 148–53.

Villegas, C. 1964. “On Qualitative Probability /sigma-Algebras.” The Annals of Mathematical Statistics 35 (4): 1787–96.

^{^}
Imprecise probabilities are a particular (well-studied) formal model of indeterminate beliefs as a set of distributions. But, as I’ll discuss, beliefs need not be modeled this way in order to have the key qualitative properties of imprecise probabilities. So I’ll use the term “indeterminate beliefs” here, following Steele (2007) and Hájek and Smithson (2012).
^{^}
Cf. a more rigorous analysis in Eriksson and Hájek (2007). Like them, I take “degree of belief” to be a primitive, in the philosophy jargon.
^{^}
As examples of rejecting the separation between beliefs and decisions/preferences, see Wei Dai here, or Karnofsky here (emphasis his): “It is important to note that I emphasize “better decisions” and not “correct beliefs”: it is often the case that one reaches a decision using cluster thinking without determining one’s beliefs about anything (other than what decision ought to be made).”
Even if you don’t consider beliefs as primary to preferences and decisions, upon reflection you might still endorse incomplete (i.e., indeterminate) preferences. Or you might think that reasonable decision-making has the properties of indeterminacy, unpacked below. If you think of Bayesian priors as “caring measures” used in Solomonoff induction (more on this in the appendix), this doesn’t imply you have a single determinate prior, or that you can’t reflect on the principles you want it to satisfy.
^{^}
See Easwaran (2014) and Meacham and Weisberg (2011) for more.
^{^}
Examples (emphasis mine):
- Shulman: “If the argument from cluelessness depends on giving that kind of special status to imprecise credences, then I just reject them for the general reason that coarsening credences leads to worse decisions and predictions.”
- Lewis: “My principal interest is the pragmatic one: that agents like ourselves make better decisions by attempting to EV-maximization with precisification than they would with imprecise approaches.” And, “Withholding judgment will do worse if, to any degree, my tentative guesswork tends towards the truth.”
- Chappell: “[I]t seems like the defense of EV against simple cluelessness could carry over to defend meta-EV against complex cluelessness? E.g. in the long run (and across relevant possible worlds), we’d expect these agents to do better on average than agents following any other subjectively-accessible decision procedure.”
I don’t think any of these claims can be interpreted as saying that acting according to indeterminate beliefs leads to sure losses, which wouldn’t beg the question, but would be false anyway.
^{^}
Or doing research, community building, or other activities that could have that side effect.
^{^}
For example, he writes: “As a bounded Bayesian, I have all the behaviors recommended by those advocating Knightian uncertainty. I put high value on increasing my hypothesis space, and I often expect that a hypothesis will come out of left field and throw off my predictions. I’m happy to increase my error bars, and I often expect my credences to vary wildly over time. But I do all of this within a Bayesian framework, with no need for exotic ‘immeasurable’ uncertainty.”
^{^}
See appendix for a more precise explanation and justification of this claim. When I say we “allow the likelihood of one hypothesis compared to another to be indeterminate”, this is equivalent to rejecting the assumption that we have a complete order over the likelihoods of hypotheses.
^{^}
Cf. Rinard (2017) and Lyon (2017), who discuss imprecise probabilities as a better representation of vague beliefs than precise probabilities, while still being imperfect.
^{^}
See, e.g., Schoenfield (2012) and Thornley.
^{^}
Imagine you’d instead known the letter would have 40,000 signatures in the first place. Presumably you would’ve also thought, “I have no clue.” What difference does it make that the weak consideration became salient to you later?
You might wonder, exactly how non-mild does the sweetening have to be to change your verdict? For an ideal agent, the principled cutoff point would be “enough evidence to update all the distributions in your representor to p(pausing AI decreases x-risk) ≥ 0.5.” But for humans, this cutoff will be vague. This is no more arbitrary than the particular cutoff for your determinate credence to exceed 0.5.
^{^}
(Cf. footnote 1 of this post.) It’s a bit philosophically tricky to avoid cluelessness “infecting” the calculus under moral uncertainty (MacAskill 2013). I currently think this can be avoided in a satisfactory way either by an approval voting approach (Elkin 2024), or with another proposal my colleagues and I are currently investigating.
^{^}
Briefly, for each money pump against indeterminate beliefs (i.e., for the axiom of completeness), either:
- The agent does not strictly prefer to take actions such that they get money-pumped; or
- The money pump can be avoided via the principle of “wise choice” (Rabinowicz 2020), which I think is a plausible way to reject so-called decision-tree/dynamic separability (see, e.g., Gustafsson (2022, p. 9)).
More controversially, though I’m personally quite sympathetic: If we endorse “time-slice rationality” (Hedden 2015), we could deny that diachronic money pumps have any force in general.
^{^}
See Bradley (2017, chap. 11) for a defense of imprecise Bayesianism. The SEP articles for imprecise probabilities and epistemic utility arguments discuss key objections; see Bradley and Steele (2014a), Bradley and Steele (2016), Konek (2019), and Bradley (2022) for responses.
^{^}
Despite raising the above objection himself, Lewis notes: “[I]f, in fact, one’s uncertainty is represented by a set of credence functions, and, in fact, one has no steers on the relative plausibility of the elements of this set compared to one another, then responding with indeterminacy when there is no consensus across all elements seems a rational response.”
^{^}
Maybe we could put equal third-order weight on each way of assigning equal second-order weights, and so on. But then our answer depends on which methods of assigning second-order weights occurred to us, out of indefinitely many methods.
^{^}
Indeed, in Superforecasting (p. 243) Tetlock writes, “[T]here is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious… These limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems. In my EPJ research, the accuracy of expert predictions declined toward chance five years out.” (H/t Kokotajlo here.)
^{^}
See Stefánsson (2017) and Bradley (2017, chap. 4) for more discussion of these distinctions.
^{^}
Villegas (1964); see also the exposition in Bradley (2017, chap. 5).
^{^}
The requirement is “minimal coherent extendability”. A “coherent extension” is a complete and transitive set of comparative beliefs that’s consistent with the complete parts of the original beliefs. “Minimality” is a technical condition that prevents it from being too easy to have a coherent extension.
^{^}
See Bradley and Steele (2014b) and references therein for discussion of alternate updating rules for imprecise probabilities.
^{^}
See also, e.g., Hájek (2004).
^{^}
Since the set of powers of side lengths is unbounded, we can’t form a uniform prior over it. So we can’t resolve this problem by aggregating the answers given by the different parameterizations. And if we try to aggregate the set of non-uniform priors, we get an infinite regress.
^{^}
See Huemer (2009) and Climenhaga (2020) for discussions of this view.
^{^}
One possibility, whose precise explanation is out of scope, is that we could ground out Occam’s razor in the principle of indifference as follows: First, our prior puts equal weight on each high-level theory, e.g. an ontology. Then, more intuitively complex high-level theories consist in a larger number of specific hypotheses, because a larger number of parameter values for the properties of the fundamental objects are possible in the given theory. So, specific complex hypotheses have lower weight because the prior mass of the corresponding high-level theory is spread more thinly. See, e.g., Henderson (2014), Rasmussen and Gharamani (2000), Huemer (2016), and Builes (2022) for related ideas.