Spurious Counterfactuals are spuriously low evaluations of the quality of a potential action, which are only provable because they are self-fulfilling (usually due to Lob's theorem). For example, if I know that I go left, then it is logically true that if I went right, I would get -10 utility (because in classical logic, false statements imply any statement). This suggests that if I fully believed that I went left, then I would indeed go left. By Lob's theorem, I indeed go left.
Building agents who avoid this line of reasoning, despite having full access to their own source code and the ability to logically reason about their own behavior, is one goal of Embedded Agency.
Spurious Counterfactuals are spuriously low evaluations of the quality of a potential action, which are only provable because they are self-fulfilling (usually due to Lob's theorem). For example, if I know that I go left, then it is logically true that if I went right, I would get -10 utility (because in classical logic, false statements imply any statement). This suggests that if I fully believed that I went left, then I would indeed go left. By Lob's theorem, I indeed go left.
Building agents who avoid this line of reasoning, despite having full access to their own source code and the ability to logically reason about their own behavior, is one goal of Embedded Agency.