Meet the Bayesians

In one way of looking at Bayesian reasoners, there are a bunch of possible worlds and a bunch of people, who start out with some guesses about what possible world we're in. Everyone knows everyone else's initial guesses. As evidence comes in, agents change their guesses about which world they're in via Bayesian updating.

The Bayesians can share information just by sharing how their beliefs have changed.

"Bob initially thought that last Monday would be sunny with probability 0.8, but now he thinks it was sunny with probability 0.9, so he must have has seen evidence that he judges as 4/9ths as likely if it wasn't sunny than if it was"

If they have the same priors, they'll converge to the same beliefs. But if they don't, it seems they can agree to disagree. This is a bit frustrating, because we don't want people to ignore our very convincing evidence just because they've gotten away with having a stupid weird prior.

What can we say about which priors are permissible? Robin Hanson offers an argument that we must either (a) believe our prior was created by a special process that correlated it with the truth more than everyone else's or (b) our prior must be the same as everyone else's.

Meet the pre-Bayesians

How does that argument go? Roughly, Hanson describes a slightly more nuanced set of reasoners: the pre-Bayesians. The pre-Bayesians are not only uncertain about what world they're in, but also about what everyone's priors are.

These uncertainties can be tangled together (the joint distribution doesn't have to factorise into their beliefs about everyone's priors and their beliefs about worlds). Facts about the world can change their opinions about what prior assignments people have.

Hanson then imposes a pre-rationality condition: if you find out what priors everyone has, you should agree with your prior about how likely different worlds are. In other words, you should trust your prior in the future. Once you have this condition, it seems that it's impossible to both (a) believe that some other people's priors were generated in a way that makes them as likely to be good as yours and (b) have different priors from those people.

Let's dig into the sort of things this pre-rationality condition commits you to.

Consider the class of worlds where you are generated by a machine that randomly generates a prior and sticks it in your head. The pre-ration

...