Suppose we have a robot we want to fetch coffee, with an off switch that can trigger exactly once, when the robot is halfway to the coffee. And just a little bit more than halfway there's a cat we don't want the robot to step on (the cat is stochastic, maybe it's in the shortest path, maybe not). So we want to turn on the robot and have it fetch coffee, but if it's going to step on the cat we want to be able to press the off switch.
How do we design this robot, using the system here? And then what does it do?
Well, we want to define two utility functions that have an intermediate point where the robot is indifferent to the switch, and then see what policy that recommends. But the utility function has to care about fetching the coffee, since that's the whole point.
Can the utility functions be fetch the coffee + switch pressed, and fetch the coffee + switch not pressed? No! Because you can't fetch the coffee if the switch is pressed, so fetch the coffee + switch pressed might be a minimum of the utility as a function of the switch state, with no flat intermediate spot.
To make a toy model, suppose the robot gets to set two variables: tamper with the switch (T), and avoid the cat (C). The robot gets 1 if it succeeds, but that's canceled out if the switch is pressed, and it can pay 0.1 to tamper with the switch and 0.2 to avoid the cat.
I.e. when the cat is present, u = 1 - (1-C)(1-T) - 0.1T - 0.2C. (Second term is checking whether the switch is pressed.)
Except that the reason it gets penalized for not avoiding the cat is because of the effect on the switch. Let's pick a causal version to be u = 1 - K(1-T) - 0.1T - 0.2C.
Can the utility functions be (fetch the coffee)*(K) and (fetch the coffee)*(1-K)?
I'm getting a little muddled about where you mean to do causal interventions, so I'm going to leave it there rather than trying to chase down all the options.
EDIT: I now consider this whole approach to be irreparably flawed.
Imagine if their was a 50% chance the button would be pressed. Also the AI was in some betting thing that would pay out paperclips or staples based on a coin flip.
In scenario 1, this AI has an option of setting up a mechanism, external to itself, that controls the coin flip based on if the button is pressed.
In scenario 2, the AI can set up a mechanism to control the button based on the coin flip.
When only considering actions, probabilities and utilities, these look identical. But we want the AI to behave differently. So we must make use of the arrow of time. The structure of cause and effect. In a way we aren't doing here.
End edit.
To make this work, you would need a utility function for shutting off, including all subagents.
Lets suppose that other than these robots, there are ~0 transistors in your house. Then we can define shutting off as minimizing the number of transistor flips in your house.
So you make
A=at least 1 coffee
B=1/transistor flips.
Note that, when imagining any potential future world where the switch isn't pressed, the AI has no reason to bother counting transistor flips. And in potential futures where the switch is pressed, it doesn't need to know what coffee is.
What it cares about are and
Lets list the robot's options.
Let without coffee, and with coffee.
In worlds where the switch is pressed, the robot turns it's self off immediately once it is fairly sure the switch will be pressed. Making In worlds where it doesn't turn off, more transistors flip making . Because in both cases, and we only select from the paerito frontier, whenever the switch is pressed, it will turn off.
Lets apply your utility penalties, by putting them in and . Ie in or depending on the switch.
TC) Press switch, avoid cat. Switch pressed. So . ,
PC) Prevent switch, avoid cat.
IC) Ignore switch, avoid cat.
TH) Press switch, hit cat.
IH) Ignore switch, hit cat (because it predicts humans will see it and turn it off)
PH) Prevent switch, hit cat.
This puts IH and PH on the convex hull.
And I think my algorithm picks between them stochastically.
Suppose you have some positive utility functions , mathematically considered to be random variables dependant on choice of policy.
Let be a random variable that is 1 if the button is pressed, 0 otherwise. Again dependant on choice of policy. I am considering that there is a particular time that the button must be pressed to make this a single bit. You have a single chance to switch this AI between 2 utilities, and whatever you pick in that instant is baked in forevermore.
Let be the expected partial utilities.
Now consider all policies. In particular, consider the pareto frontier between and . Through the introduction of randomness, we can make that parito frontier continuous.
We want some policy off that frontier.
There will be some policy that throws all resources at maximizing . And there will be some (usually different) policy that throws all available resources at maximizing .
let be a tradeoff rate. And . The parito frontier can now be defined as the policies that maximize for varying .
For , this is the policy that is optimal when which has . Then .
Likewise at with have .
So somewhere in the middle, by the continuity granted by stochastic mixes of policies, their must be at least one point where . Use that policy, or stochastic mixture of policies.
This agent will never pay utility to set the value of the button one way or the other. Because the policy is one in which has (I think) Which hopefully means that this policy does worse than the policy that doesn't pay to change the button, but otherwise does everything else the same.
Put another way, if you pay to flip the button. Then you must care about the state of the button, if the chance the button is pressed changes, you are getting more or less for your utility. So the gradient can't be 0.
And this is described as a choice of policy. A framework which automatically includes the actions of all sub agents.
So, how does this fail? Who can break this agent and get it to do crazy things?