Scientist by training, coder by previous session,philosopher by inclination, musician against public demand.
Team Piepgrass: "Worried that typical commenters at LW care way less than I expected about good epistemic practice. Hoping I’m wrong."
Corrigibility has various slightly different definitions, but the general rough idea is of an AI that does what we want
An aligned AI will also so what we want because it's also what it wants, its terminal values are also ours.
I've always taken "control" to differ from alignment in that it means an AI doing what we want even if it isn't what it wants, ie it has a terminal value of getting rewards, and our values are instrumental to that, if they figure at all.
And I take corrigibility to mean shaping an AIs values as you go along and therefore an outcome of control.
More generally, the problem is that for formal agents, false antecedents cause nonsensical reasoning
No, it's contradictory assumptions. False but consistent assumptions are dual to consistent-and-true assumptions...so you can only infer a mutually consistent set of propositions from either.
To put it another way, a formal system has no way of knowing what would be true or false for reasons outside itself, so it has no way of reacting to a merely false statement. But a contradiction is definable within a formal system.
To.put it yet another way... contradiction in, contradiction out
This statement of the problem concedes that SB is calculating subjective probability. It should be obvious that subjective probabilities can diverge from each and objective probability -- that is what subjective means. It seems to me that the SB paradox is only a paradox if y ou try to do justice to objective and subjective probability in the same calculation.
Of you course, Hawkins doesn't just say they are stupid. It is Byrnes who is summarily dismissing Hawkins, in fact.