Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents.

I've given examples of preferences over non-rewards.

But those examples are easy to dismiss as irrational or imposing on others, as they involved religion, or things like preferences over other people knowing the truth.

But if we think in terms of preferences over personal identity, we have far more widespread and natural examples of this.

One small example: in most murder-mysteries, I'd prefer not to know the identity of the killer at the beginning of the show. If I'm taking part in a murder mystery dinner, I'd definitely not want too. If I'm playing games of hidden information for fun, then I would not want to know all the hidden information - or else that makes this pointless. Lots of people enjoy watching sport events live; much less people re-watch random past sporting events whose result they already know. Eliezer talks about the joy in scientific discovery.

So preferences over non-rewards are quite natural and common.

Hedonism solution?

Now, you could say you don't have a "preferences over knowing/not knowing something", but that you "enjoy the experience of learning", and thus reduce the whole thing to hedonism. Eliezer argues against this, claiming to have preferences that don't reduce to hedonism (preferences I share, incidentally).

But even fully giving in to hedonism only partially solves the problem; yes, we can model someone's faith as saying they enjoy the experience of belief, and murder-mystery watchers enjoy the experience of seeing intricate puzzles solved. But this doesn't solve the problem that, to maximise hedonic experience, we still have to control what knowledge the human has.